Mechanism for the capture of graphical representations

ABSTRACT

A visual link mechanism residing in a local system for identifying addresses of locations in the plurality of remote systems wherein the local system is connected through a network to the plurality of remote systems. The visual link mechanism includes a visual link library and a network access mechanism responsive to a visual link including a displayable graphic icon for accessing the location represented by a selected graphic icon. Various structures of visual links are described, each being an entity existing independently of the system environment in which it resides, and the network access mechanism includes a layout table for storing a plurality of plans for arranging and displaying a plurality of visual link graphic icons in a display, a visual links organizer, a visual link screen saver, and a hash protection mechanism for detecting the unauthorized construction or modification of visual links or other forms of files. Also described is a visual link capture engine for extracting graphics information from a data file and generating a corresponding graphic icon and a display layout generator for generating display layouts of sets of predetermined numbers of displayable visual objects.

CROSS REFERENCES TO RELATED APPLICATIONS

The present Patent Application is a Divisional Patent Application ofco-pending Prior patent application Ser. No. 08/814,118, filed Mar. 10,1997 and which is incorporated herein by reference and is related toU.S. Patent Appellation Serial No. by Kenneth Charles Knowlton and GarySteven Miliefsky for A PROTECTION MECHANISM FOR VISUAL LINK OBJECTS andto U.S. patent application Ser. No. 09/302,344 by Kenneth CharlesKnowlton and Gary Steven Miliefsky for A DISPLAY LAYOUT GENERATOR FORGRAPHICAL REPRESENTATIONS, both of which are filed on even date with thepresent Patent Application.

FIELD OF THE INVENTION

The present invention relates to a method and apparatus for thegraphical representation of bodies or sources of information and, inparticular, a method and apparatus for capturing and displaying graphicrepresentations of bodies or sources of information and for indexing andaccessing the bodies or sources of information.

BACKGROUND OF THE INVENTION

The volume of information accessible to users of computer systems,either directly on a local system or indirectly through a network, suchas the internet and the World Wide Web constructed on the internet,usually referred to as “the Web”, has increased to the point wheretracking and indexing the information so that the user can locate agiven body or source of information has become a primary problem. Forexample, users may have literally thousands of documents, spreadsheets,and graphics files directly available on their local system and as manyuniversal resource locators (URLs), wherein each URL essentially pointsto or is an address of a location of a body or source of information onthe internet and contains the information required to direct a TCP/IPbased protocol executing on a system to a particular location orresource on the internet and on intranets.

The problem is compounded in that the methods and means in general usefor indexing such information remains relatively primitive compared tothe methods and means for generating and accessing the information.

For example, it is well known and widely accepted that humans comprehendcomplex or voluminous information more readily when it is presented ingraphical form rather than as text or numbers. This characteristic is,in fact, widely exploited in computer operating environments such asMicrosoft Windows and Apple Macintosh and in most applications programs,such as Netscape Navigator, and even in much of the information itself,which is frequently graphical in nature or uses or incorporatesgraphical information.

The methods presently in use for indexing information or sources ofinformation, however, generally represent bodies or sources ofinformation, such as files or URLs, in text form, that is, as lists ofalphanumeric names or designators. For example, even the computeroperating environments and applications programs providing graphicaluser interfaces, such as Microsoft Windows, the Apple Macintosh andNetscape Navigator, either use alphanumeric designators alone or usealphanumeric designators to distinguish between otherwise identicalicons representing files generically.

The systems of the prior art have generally retained alphanumericdesignators to represent bodies or sources of information because of thedifficulties inherent in representing each individual file or source ofinformation with a graphic representation that is sufficientlydistinguishable from other graphic representations of files or sourcesof information and that presents a visually strong image to a user.

The primary problem in this respect is the volume of data that must bestored to represent each file or source of information individually whenthere are potentially hundreds of such files or sources of informationto individually designate. That is, it has been generally accepted inthe prior art that it is necessary to have a relatively large andcomplex graphic representation of each file or source of information inorder to have visually strong and distinguishable representations offiles or sources of information when there is more than a relativelylimited number of files or sources of information. As such, only a fewspecialized applications programs, such as the Corel graphics programsand, for example, hypertext markup language (HTML) pages, such asfrequently used on the Web, have attempted to provide any form ofgraphically based indexing of files. These programs and HTML pages havegenerally used compressed versions of full size images for this purpose,which are in themselves large graphics files, thereby requiring longextensive storage space and long transmission times to transfer over anetwork.

Related problems are, of course, the overwhelming effort necessary togenerate what may be literally hundreds of unique representations to beassociated with various files and sources of data, and the inherentproblems in organizing and displaying such graphic representations forfiles and information sources on a typical display.

Still other related problems are, for example, the detection of theunauthorized construction or modification of computer files, such asvisual links, and an effective method for providing encryptedauthorization within a file, again such as a visual link, withoutincreasing the size of the file or causing difficulties for human andsystem readers of the file.

The present invention provides a solution to these and other problems ofthe prior art.

SUMMARY OF THE INVENTION

The present invention is directed to a visual link mechanism residing ina local system for identifying addresses of locations in the pluralityof remote systems wherein the local system is connected through anetwork to a plurality of remote systems and typically includes a memoryfor storing data and programs, a processor operating under control ofthe programs to perform operations on the data, and a display.

According to the present invention, the visual link mechanism includes avisual link library for storing visual links wherein each visual linkcorresponds to an address of a location in the plurality of remotesystems, and a network access mechanism connected to the network fordisplaying the visual links to a user and allowing the user to select avisual link representing a corresponding location in a remote system.The network access mechanism is responsive to the visual link selectedby the user for accessing the location represented by the selectedvisual link, and each visual link includes a graphic icon providing adisplayable image representing the corresponding location and theaddress of the corresponding location.

Each visual link may also include alphanumeric information relating tothe corresponding location, including a title identifying the locationand text describing the location, and may include keywords relating tothe contents of the location.

Each visual link preferably may include the graphic icon providing adisplayable image representing the corresponding location and a visuallink dataset containing information relating to the location, includingthe address of the corresponding location, wherein each visual linkdataset may include a plurality of designators, each designator storingcorresponding information relating to the corresponding location andincluding a graphic icon designator for storing an identification of thecorresponding graphic icon, an address designator for storing theaddress of the corresponding location, and an end designator forindicating the end of the visual link dataset.

The designators may further include a subject designator for storingtext information identifying the contents of the location, an authordesignator for storing text information identifying the author of thecontents of the location, and a keyword designator for storing key wordsrelating to the contents of the location.

According to the present invention, the visual link mechanism furtherincludes a link for relating the graphic icon and the visual linkdataset. In one implementation, the link comprises a file for storing apathname of the graphic icon and a pathname of the visual link datasetand in another the link comprises a graphic icon pointer designatorstored in the visual link dataset for storing an identification of thegraphic icon. In still other implementations, the graphic icon and thevisual link dataset are stored in a single file, the graphic icon beingstored starting at the beginning of the file and the visual link datasetfollowing the graphic icon, wherein the visual link dataset furtherincludes a length of dataset number stored at the end of the file andindicating the location of the start of the visual link dataset relativeto the end of the file. In yet other implementations, the visual linkdataset further includes an address pointer and an end pointer storedpreceding the length of dataset number and indicating the locations ofthe address designator and the end designator in the file.

Further according to the present invention, the network access mechanismincludes a layout table for storing a plurality of plans for arrangingand displaying a plurality of graphic icons in display spaces whereineach plan corresponds to a display space of predetermined dimensions andto a predetermined number of graphic icons of predetermined dimensionsand defining the arrangement of the graphic icons in the display space,a input for providing a set of display definition inputs wherein eachset of display definition inputs includes a display space definitionidentifying the dimensions of a display space and a set of visual objectdefinitions identifying a set of graphic icons of a predetermined numberand of predetermined dimensions to be displayed in the display space. Adisplay generator is responsive to the display definition inputs forreading from the layout memory a plan corresponding to a set of displaydefinition inputs and generating a display of the identified set ofgraphic icons in the display space according to the corresponding planand displaying the identified set of graphic icons on the display. Inone implementation, the input for providing a set of display definitioninputs includes a visual link display memory for storing groups ofidentifications of visual links, each group of identifications of visuallinks being a set of visual object definitions identifying a set ofgraphic icons of a predetermined number and of predetermined dimensionsto be displayed in the display space.

Still further according to the present invention, the network accessmechanism includes a visual links organizer connected to the display fordisplaying the visual links to a user and allowing the user to select avisual link representing a corresponding location in a remote system anda network communication mechanism connected to the visual linksorganizer and responsive to the visual link selected by the user foraccessing the location represented by the selected visual link.

The visual link mechanism of the present invention may also include avisual link screen saver connected from the visual link library and tothe display and responsive to operations of the local system forgenerating a visual link screen saver display of selected graphic iconson the display wherein the screen saver is responsive to an indicationof a displayed graphic icon of the visual link screen saver display forinvoking the a network access mechanism for accessing the locationrepresented by the selected visual link corresponding to the selectedgraphic icon.

In other aspects, the present invention includes a hash protectionmechanism for detecting the unauthorized construction or modification ofvisual links or other forms of files, including a file reader forreading an encryption part of a file and a stored hash encryption valuestored in a storing part of the file, a hashing mechanism for generatingan encrypted hash value from the data contained in the encryption partof the file, and a comparator for comparing the generated encrypted hashvalue and the stored hash value and determining when the generated andstored hash values are equal. According to the present invention, thestoring part of the file includes data of the file separate from theencrypting part of the data of the file and the data of the storing partof the file is characterized in having alternate and semanticallyequivalent expressions wherein the stored hash encryption value isencoded in the data of the storing part of the file using the alternateand semantically equivalent expressions representing the data of thestoring part of the file to encode the stored hash encryption value.

The present invention also includes a capture engine for extractinggraphics information from a data file and generating a correspondinggraphic icon forming a displayable image representing the graphicsinformation. According to the present invention, the capture engineincludes a grayscale image generator for receiving an original image andgenerating a corresponding grayscale image containing brightness valuesrepresenting the original image, an edge image mechanism for receivingthe grayscale image and generating a corresponding edge imagerepresenting areas of visually significant graphic structure in thegrayscale image as represented by areas of change in the brightnessvalues, a candidate region search mechanism for receiving the edge imageand identifying initial candidate regions of the edge image representingvisually significant areas of the original image, and a candidate regionadjustment and comparator mechanism for selecting a candidate region tobe used in generating a corresponding graphic icon and for adjusting theselected candidate region to conform to predetermined dimensions for agraphic icon.

In further aspects of the present invention, the grayscale imagegenerator includes a grayscale lookup table for storing grayscalebrightness values corresponding to each of the possible brightnessvalues of image elements in original images, a grayscale converter forreading the brightness values of the image elements of the originalimage, reading the corresponding grayscale brightness values stored inthe grayscale lookup table, and generating a corresponding grayscaleimage wherein each image element of the original image is represented bya grayscale image element having the grayscale brightness valuecorresponding to the brightness value of the image element.

Also according to the present invention, each image element of theoriginal image is represented by a set of color values, the set of colorvalues of an image element of the original element representing thecolor and brightness value of the image element. Accordingly, thegrayscale lookup table contains a grayscale brightness valuecorresponding to each of the possible sets of color values of imageelements in original images and the grayscale converter is responsive tothe set of color values of each image element of the original image forreading the corresponding grayscale brightness values stored in thegrayscale lookup table and generating a corresponding grayscale imagewherein each image element of the original image is represented by agrayscale image element having the grayscale brightness valuecorresponding to the set of color values of the image element.

Also, the grayscale image generator generates each image element of thegrayscale image from a plurality of corresponding elements of theoriginal image so that the number of image elements in the grayscaleimage is proportionally reduced from the number of image elements in thecorresponding original image.

According to the present invention, the edge image mechanism includes anedge image generator for receiving an input grayscale image andgenerating a corresponding output edge image wherein each image elementof an output edge image is represented by an edge value proportionate tothe difference between the brightness value of the corresponding imageelement in the input grayscale image and the brightness values of theneighboring image elements of the corresponding image element in theinput grayscale image. An output edge image thereby represents areas ofvisually significant graphic structure in the input grayscale image asrepresented by areas of change in the brightness values of the inputgrayscale image.

The edge image mechanism may also include a lowpass filter connectedbetween the grayscale image generator and the edge image generator forreceiving a grayscale image and generating a corresponding filteredgrayscale image of reduced resolution and provides the filteredgrayscale image to the edge image generator as the correspond inputgrayscale image to the edge image generator, wherein each image elementof a filtered grayscale image is determined as a proportioned summationof the brightness values of selected neighboring image elements of theimage element of the grayscale image so that a filtered grayscale imagethereby represents visual structures of significant extents of thegrayscale image.

In further aspects of the present invention, the edge image generatorgenerates each image element of an output edge image from a plurality ofcorresponding elements of the input grayscale image so that the numberof image elements in the output edge image is proportionally reducedfrom the number of image elements in the corresponding grayscale inputimage.

Still further according to the present invention, the candidate regionsearch mechanism includes a rectangular sum array generator forreceiving an output edge image and generating a correspondingrectangular sum array having a sum element corresponding to each imageelement of the output edge image wherein each sum element represents thesum of the edge values of the image elements of a region of the outputedge image bounded by a coordinate location of the output edge image andthe output edge image element corresponding to the sum element. Arectangular sum search mechanism is connected from the rectangular sumarray generator to receive the rectangular sum array for examining thesummed edge values represented in each of the sum elements of therectangular sum array and identifying at least one initial candidateregion having a high summed edge value, each initial candidate regionthereby representing and corresponding to a region of the original imagehaving a significant visual structure.

Still further, the candidate region adjustment and comparator mechanismincludes an edge adjuster for receiving the at least one initialcandidate region and generating from each initial candidate region acorresponding grown candidate region by adjusting the extents of eachinitial candidate region to include all portions of significant visualstructures having a part thereof contained within the initial candidateregion, generating from each grown candidate region a corresponding edgeadjusted candidate region by adjusting the extents of each growncandidate region so that each edge adjusted candidate region conforms toone of a set of predetermined extents for graphic icons, comparing eachedge adjusted candidate region with others of the edge adjustedcandidate regions and selecting the edge adjusted candidate regionhaving the highest amount of significant visual structure, and comparingeach selected edge adjusted candidate region with previously existinggraphic icons. When the selected edge adjusted candidate region issimilar to a previously existing graphic icon, the edge adjuster selectsa next one of the at least one edge adjusted candidate regions, and,when a selected candidate region is distinguished from the previouslyexisting graphic icons, identifies the corresponding region of theoriginal image for use as a graphic icon.

The present invention also provides a display layout generator forgenerating layouts of sets of predetermined numbers of displayablevisual objects of predetermined dimensions into a display area ofpredetermined dimensions. According to the present invention, thedisplay layout generator includes a possible layouts generator forreceiving a display space definition defining the dimensions of thedisplay area and a visual object definition defining the numbers anddimensions of the visual objects to be arranged in the display area andgenerating all possible layouts of the visual objects in the displayarea by placing in a layout a visual object and searching forward with anext visual object to identify a placement of the next visual object andbacktracking on each next placed visual object by removing the nextplaced visual object and selecting and placing a different next visualobject until all combinations of visual objects and placements of visualobjects have been exhausted.

The display layout generator further includes a layout memory forstoring each arrangement of the set of visual objects in the displayarea generated by the possible layouts generator, a layout testmechanism for testing each new layout by comparing each new layout withpreviously existing layouts to determine when the new layout has asubpopulation of a previously existing layout or matches a previouslyexisting layout and when the previously existing layout has asubpopulation of the new layout, wherein a subpopulation is a subset ofthe visual objects, a layout previous/next selector for selecting thenew layout or the previously existing layout by discarding the newlayout when the new layout has a subpopulation of a previously existinglayout or matches a previously existing layout and replacing thepreviously existing layout with the new layout when the previouslyexisting layout has a subpopulation of the new layout, a layoutscomparator for comparing pairs of layouts stored in the layout generatormemory and discarding any layout that contains a subpopulation ofanother layout, and a layout table for storing the final layouts for usein generating visual displays of the sets of visual objects in thedisplay areas.

According to the present invention, the layout generator may furtherinclude a screen map generator for receiving the display spacedefinition and the visual object definition and generating a screen mapcorresponding to the defined display area and having instance locationsrepresenting possible placement locations of the visual objects in thedisplay area, an instance pattern generator for receiving the screen mapand the visual object definition and generating, for each type of visualobject of the set of visual objects wherein a type of visual object isdefined by the dimensions of a visual object, at least one instancepattern representing all possible instances of locations of the visualobject in the screen map, and an instance pattern enumerator forenumerating the resulting instance patterns and providing the enumeratedinstance patterns to the possible layouts generator to be used asinitial starting points in generating layouts.

The layout generator may further include a screen spatial divisiongenerator connected from the possible layouts generator for receivingthe display space definition and the enumerated instances patterns anddividing the defined display area into spatial divisions along axes ofsymmetry, a location instances sorter for sorting, for each spatialdivision of the display area, the enumerated instance patterns of visualobjects in each spatial division by visual object type, and generating aset of location instances of visual objects for each spatial division, alocation instance search and compose mechanism for searching for andgenerating, from the sets of location instances of visual objects foreach spatial division, at least one combination of location instances inat least one spatial division wherein each combination of locationinstances contains a combination of visual objects of the set of visualobjects, and a super-object generator for designating each combinationof visual objects as a visual super-object and providing thedesignations to the possible layouts generator as respresentingcorresponding visual object.

The layout generator may also include a comer area examiner connectedfrom the possible layouts generator for receiving the layouts andexamining the comer areas of the layouts for unique combinations ofvisual objects and discarding layouts having comer areas having layoutsof visual objects that are rotational or reflectional equivalents of apreviously existing layout.

Finally, the layout generator may also include a screen sub-area dividerconnected from the possible layouts generator for receiving the displayarea definition and dividing the display area into sub-areas and provideto the possible layouts generator a display space definition for eachsub-area for use in generating layouts for each sub-area and a screensub-area combiner connected from the layout comparator for composinglayouts for the display area by combining the layouts for the sub-areasand provide the combined layouts to the layout comparator as the layoutfor the defined display area.

DESCRIPTION OF THE DRAWINGS

The foregoing and other objects, features and advantages of the presentinvention will be apparent from the following description of theinvention and embodiments thereof, as illustrated in the accompanyingfigures, wherein:

FIG. 1A is a block diagram of a system embodying the present invention;

FIG. 1B is diagrammatic representations of a database entry and a visuallink;

FIG. 1C illustrates the embedding of a visual link reference in ahypertext document;

FIG. 1D illustrates the designation of an area of a hypertext documentto comprise a graphic icon;

FIG. 2A is a diagrammatic representation of the image data structuresgenerated and used in capturing a graphic icon;

FIG. 2B illustrates the arrangement or tiling of graphic icons withborders in a display;

FIGS. 3A-1 and 3A-2 are block diagrams of a capture engine of thepresent invention;

FIGS. 3B, 3C, 3D, 3E and 3F are diagrammatic representations of scalingand edge determination performed by the capture engine of the presentinvention;

FIGS. 3G and 3H illustrate the use of a rectangular sum array todetermine area content by the capture engine of the present invention;

FIG. 4A is a block diagram of a visual links organizer for generatingdisplay layouts and plans for graphic icons;

FIGS. 4B, 4C and 4D are illustrations of graphic icon layouts and plans;

FIGS. 4E-1 through 4E-5 are block diagrams illustrating the generationof graphic icon layouts and plans by the visual links organizer;

FIGS. 4F, 4G, 4H, 4I, 4J, 4K, 4L, and 4M are illustrations of datastructures and layouts used in the operation of the visual linksorganizer in generating graphic icon layouts and plans;

FIGS. 5A through 5G illustrate embodiments of visual links according tothe present invention; and,

FIGS. 6A and 6B illustrate a hashing protection mechanism for detectingthe unauthorized construction or modification of a file such as a visuallink.

DESCRIPTION OF THE INVENTION

As described above, the present invention is a system and a method andapparatus for the graphical representation of bodies or sources ofinformation, such as files or internet or Web locations, and, inparticular, for capturing and displaying graphic representations of thebodies or sources of information for use in indexing and accessing thebodies or sources of information. The following will first describe apresently preferred embodiment of a system for indexing and displayingand accessing information on the World Wide Web, hereafter referred toas the Web, and will then describe components of that system forcapturing and displaying graphic representations of bodies ofinformation, such as files, or sources of information, such as sites orHTML documents, on the Web and the use of such graphical representationsto access such bodies or sources of information. It will be appreciatedby those of ordinary skill in the arts, however, that the present systemfor indexing and displaying and accessing information accessible acrossor through a network is not limited to the World Wide Web and HTMLdocuments residing therein, but is also applicable for indexing anddisplaying and accessing information accessible across or through otherforms of networks, such as electronic mail systems, wherein a locationon such a network is identified by some form of pointer or address andwherein locations, files or persons on such a network are visuallyidentifiable to a user, for example, by means of a representativegraphic icon or, in an electronic mail system, a photograph of arecipient. The following descriptions, however, will focus on exemplaryembodiments of the present invention as implemented for use on WorldWide Web systems as representative of all generally similar types ofnetworks wherein users wish to identify, index and access locations,files or persons on the networks.

Finally, it will be noted that in the following each reference number isa three or four digit number comprised of two rightmost digits and oneor two leftmost digits wherein the one or two leftmost digits refer tothe number of the figure in which the referred to element first appearswhile the right two digits identify the element within the figure inwhich the element first appears. Once assigned, a given reference numberis used throughout the subsequent figures and descriptions to refer tothe corresponding element.

It will also be noted that closely related figures, that is, figuresillustrating related discussions in the following description, areindicated as a group of related figures by assigning a figure number tothe group of figures and distinguishing among the figures of a group byalphabetic suffixes, such as FIG. 1A, FIG. 1B, and so on. In addition,certain figures are contained on multiple drawings and, in suchinstances, the individual drawings comprising the figures and theportions of the figures appearing thereupon are indicated and referredto by a figure number with an appended drawing number; for example, aFIG. 3A may be related to FIGS. 3B and 3C and may be comprised of threedrawings identified as FIG. 3A-1, FIG. 3A-2 and FIG. 3A-3. In a similarmanner, certain figures may be optionally combined with other figures,for example, to illustrate alternate or expanded embodiments of thesubject matter illustrated in a first figure, and in these instances thefigures are again identified by a figure number with an appended drawingnumber, so that a mechanism illustrated in a FIG. 4E-1 may be extendedor expanded by additional components or mechanisms illustrated in FIGS.4E-2, 4E-3, and so on.

A. DESCRIPTION OF A PREFERRED EMBODIMENT OF THE INVENTION 1. Descriptionof a System 110 for Capturing and Displaying Bodies and Sources ofInformation, and Accessing the Information (FIGS. 1A, 1B, 1C and 1D)

Referring to FIG. 1A, therein is shown a block diagram of a System 110incorporating the present invention. The System 110 is illustratedtherein as having a Local System 112 connected to a Server System 114through a Network 116. In the present discussion, and for purposes ofillustration of an embodiment of the present invention, Network 116 isthe internet and, in particular, the World Wide Web (Web) implemented onthe internet, and Server System 114 is a World Wide Web server whileLocal System 112 and Server System 114 communicate with one another andwith other Web Servers 118 through an appropriate communicationsprotocol, such as TCP/IP.

Referring first to Server System 114, as shown in FIG. 1A Server System114 includes a Network Service System 120 for communicating with LocalSystem 112 and Web Servers 118 and a Network Data Collector and IndexingServer 122, such as a Web “Spider”, for automatically searching Network116 to locate Web Servers 118 and to access, capture and retrieveinformation stored on Web Servers 118, such as in HTML documents orother forms of files. As is well known in the relevant arts, NetworkData Collector and Indexing Server 122 operates to construct an IndexedDatabase 124 containing the information retrieved from Web Servers 118wherein the retrieved information, is represented in FIG. 1A as storedin Database Entries 126.

As illustrated in FIG. 1B, each Database Entry 126 will typicallyalphanumeric information that include a Universal Resource Locator (URL)128, often referred to as a “link”, identifying the location or addresson the Web of a corresponding HTML document, a Title 130 containing textof any arbitrary length and usually comprising a name of the HTMLdocument, a Text 132 of any arbitrary length and usually comprising abrief description of the HTML document or its contents. Each DatabaseEntry 126 may also include a Date 134 containing the date and time the“link” was last updated, and may include Keywords 136, which may containone or more keywords characterizing the HTML document and used to searchfor and locate the corresponding Database Entry 126. In otherimplementations, such as electronic mail systems, for example, the roleof URLs in the following discussions may be fulfilled by electronic mailaddresses of various forms, in a manner that will be well and easilyunderstood by those or ordinary skill in the relevant arts.

As is well known and understood in the relevant arts, a user of LocalSystem 112 may use a web program executing on Local System 112, such asNetscape Navigator, to access the Web and, with particular regard to thepresent invention, to access Server System 114 to search for Web Servers118 having information sought by the user. Server 114 will then accessand search Indexed Database 124 to locate Database Entries 126 relevantto the user's request, and will return at least portions of the DatabaseEntries 126 to Local System 112, such as the URL 128, Title 130 and Text132. The user of Local System 112 may then use the returned URL's 128 toaccess and retrieve the corresponding HTML documents from Web Servers118 and may store a URL 128 returned from Server System 114 forsubsequent use, together with its associated information. The user ofLocal System 112 may also, as is common practice, capture and store URLs128 of documents and sites and their associated information, such asTitle 130, and perhaps Text 132, from a document or site accessed on aWeb Server 118. Again, in other implementations, such as electronic mailsystems, for example, the role of a web server and index may befulfilled by electronic mail servers of various forms and electronicmail indexes residing on electronic mail servers, in a manner that willbe well and easily understood by those or ordinary skill in the relevantarts.

According to the present invention, Server System 114 further includes aVisual Links Automatic Capture Engine 138A which, as described infurther detail in the following, accesses HTML pages located in WebServers 118 by Network Data Collector and Indexing Server 122 andconstructs a Visual Links Database 140 which contains Visual Links 142corresponding to the HTML documents indexed in Database Entries 126. Asshown in FIG. 1B, and as will be discussed in further detail in asubsequent description, a Visual Link 142 will include alphanumericinformation including a Universal Resource Locator (URL) 128 and mayinclude a Title 130, a Text 132, and a Date 134, and may includeKeywords 136.

In addition, however, a Visual Link 142 includes a Graphic Icon 144which is generated or extracted from the graphics and text informationpresent in the corresponding HTML document, or other type of file, byVisual Links Automatic Capture Engine 138A and which, as described infurther detail in the following, serves as a visual representation tothe user of the associated document or file. Yet again, in otherimplementations, such as electronic mail systems, for example, the roleof Graphic Icons 144 in the following discussions may be fulfilled bygraphic representations of various forms of files, locations orrecipients on a network, such as photographs of electronic mailrecipients, in a manner that will be well and easily understood by thoseor ordinary skill in the relevant arts.

In alternate embodiments of Server System 114, for example, one whichoperates entirely with Visual Links 142, Visual Links Database 140 mayentirely replace Indexed Database 124. In more typical embodiments, forexample, until or unless Visual Links 142 are universally adopted,Server System 116 will operate with a mixture of text only links andVisual Links 142. Such systems may be implemented as described above,with matching, linked entries in Indexed Database 124 and Visual LinksDatabase 140, or with text only links stored in Indexed Database 124 andVisual Links 142 stored in Visual Links Database 142. In yet otherembodiments, Server System 114 may be implemented with the text portionsof all links in Indexed Database 124 and any corresponding Graphic Icon144 stored in Visual Links 142 with the text and graphics portionslinked to one another, or Server System 114 may be implemented with onlya Visual Links Database 140, as described above, but with the GraphicIcon 144 component of all text only links left empty.

Finally in this regard, it has been described that in certainimplementations of Server System 114, Graphic Icons 144 may be generatedby a Visual Links Automatic Capture Engine 138A residing in a ServerSystem 114 from the text and graphics information contained in each HTMLdocument, or other type of file that has been accessed by Server System114. Graphic Icons 144 may also be generated in a like manner by a userof a Local System 112, using an implementation of Visual Links AutomaticCapture Engine 138A residing in the Local System 112, or by theoperators of a Web Server 118, again using a Visual Links AutomaticCapture Engine 138A. It will also be noted that Graphic Icons 144 neednot be generated from an HTML document by a Visual Links AutomaticCapture Engine 138A, but may be generated independently by a user of aLocal System 112 or the operators of a Server System 114 or a Web Server118, or even by an independent supplier of pre-made Graphic Icons 144using, for example, virtually any of a wide range of well known andreadily available graphics creation programs, and may be usedsubsequently in the construction of Visual Links 142 or, for example, toreplace an already existing Graphic Icon 144 associated with a VisualLink 142.

In a like manner, Visual Links 142 may also be generated or otherwisecreated by the operators of Web Servers 118, by the operators of ServerSystem 114, or by the users of Local Systems 112, or otherwise provided.The Visual Links 142 so created may be embedded in the HTML documentsresiding in Web Servers 118 or in a Server System 114 or Local System112, or may be stored in indexes or databases residing in a Web Server118, a Server System 114 or a Local System 112. For example, a VisualLinks 142 may be read directly from an HTML document or other file byNetwork Data Collector and Indexing Server 122, stored in IndexedDatabase 124 or Visual Links Database 140, as described above, andprovided to and stored in a Local Server 112 from Server System 114. Infurther example, a Visual Link 142 may also be read directly by LocalServer 112 when the document or other file is accessed by Local Server112, typically by the user “clicking” on a representation of the VisualLink 142 embedded and displayed in the HTML page or file, as describedbelow, in the manner usual for HTML links.

Considering Visual Links 142 further, and assuming the use of HTMLdocuments other forms of files supporting linking, such as filesgenerated or operated upon application programs supporting MicrosoftWindows Object Linking and Embedding (OLE), a Visual Link 142 can beinserted into an HTML document by means of an OBJECT Tag or an EMBED Tag146, depending upon whether Netscape of ActiveX standards are followed.As illustrated in FIG. 1C, the OBJECT or EMBED Tag 146 follows an IMGTag 148 displaying a Visual Links Symbol 150 and is hyperlinked 152 tothe URL 154, and associated link related information, so that a user“clicking” on Visual Links Symbol 150 will access the URL 154 andassociated information. In an alternate implementation, alternatehyperlinked text or graphics, such as a Visual Link 142 may be includedwith the OBJECT or EMBED tag.

In this regard, it will be appreciated by those of ordinary skill in therelevant arts that there are a number of ways for a user to interactwith a Visual Links Symbol 150. For example, when a user passes a mousecursor over a Visual Links Symbol 150 or icon, the symbol or icon may bereplaced by the URL or by key words or by a descriptive text so long asthe cursor is colocated with the symbol or icon, or this information maybe displayed in a separate window, such as a “pop-up” window.

Referring now to Local System 112, as shown in FIG. 1A, Local System 112is, for example, a personal computer executing the Microsoft Windowsoperating environment and including typical components, such as a DiskDrive 156A and a Memory 156B for storing programs and data, a Processor158 for operating on the data under control of the programs, and aDisplay 160 for providing a visual display of the operations of thesystem. Although not shown explicitly in FIG. 1A, it will be understoodby those of ordinary skill in the relevant arts that Server System 114and Service System 116 will likewise include a Disk Drive 156A and aMemory 156B for storing programs and data and a Processor 158 foroperating on the data under control of the programs and will generallyinclude a Display 160 for providing a visual display of the operationsof the systems.

As also shown in FIG. 1A, Local System 112 further includes a Web AccessProgram 162, such as Netscape Navigator, as will Server System 114,although in a different form and implementation and capabilities fromthat in Local System 112, and, according to the present invention, aVisual Links Control 164 and a Visual Links Organizer 166, both of whichare associated with and operate in association with Web Access Program162. Associated with Visual Links Control 164 and Visual Links Organizer166 is a Visual Links Library 168 for storing Visual Links 142. LocalSystem 112 may also include a Visual Links Automatic Capture Engine 138Aor 138B, perhaps in more limited form than that implemented in ServerSystem 114, thereby allowing a user of Local System 112 to create VisualLinks 142 locally, rather than only downloading Visual Links 142 fromServer System 114 or Web Servers 118.

Visual Links Control 164 operates to interface and integrate VisualLinks 142, Visual Links Organizer 166 and Visual Links Automatic CaptureEngine 138A or 138B, if present, into Web Access Program 162, and allowsa user to access the target of a Visual Link 142 and to save a VisualLink 142 in Visual Links Library 168 or a file. Visual Links Organizer166, in turn, allows a user to create and edit Visual Links 142, tostore Visual Links 142 in Visual Links Library 168, and to otherwisemanage Visual Links 142 and Visual Links Libraries 168. Visual LinksOrganizer, which will be described in detail further below, also allowsa user to organize the display of Visual Links 142 in a Visual LinksLibrary 168.

As shown in FIG. 1A, a Local System 112 may also include a Visual LinksScreen Saver 170 which is connected from Visual Links Library 168 andwhich is responsive to the operations of Local System 112 in the usualmanner of screen savers to generate a screen saver display on Display160. The design and operation of screen savers is well known andunderstood by those of ordinary skill in the relevant arts and, as such,will not be discussed further herein. It must be noted, however, that inaccordance with the present invention the visual elements of the screensaver display are comprised of Graphic Icons 144 from Visual LinksLibrary 168, and may be selected by Visual Links Screen Saver 170 atrandom or by user control and may, for example, include the mostfrequently used Graphic Icons 144 or selected layouts of Graphics Icons144, as will be described in a following discussion. In addition, theusual operation of a screen saver utility in disabling the screen saverand returning the display presented on Display 160 to the executingoperations of the system upon, for example, any mouse input by a user,may be selectively disabled, thereby allowing the mouse to be used inconjunction is the display of Graphic Icons 144 by Visual Links ScreenSaver. This operation, in turn, allows the user to select, or “click”, aGraphic Icon 144 being displayed by Visual Link Screen Saver 170, in thesame manner as Web Access 162 and Visual Link Organizer 166, to invokeWeb Access 162 and to access the site, document or file associated withthe selected Graphic Icon 144.

Lastly, it will be noted that FIG. 1A illustrates additional elements ofthe present invention, specifically a Layout Table 412 and a DisplayGenerator 414 in Visual Links Organizer 166, which will be discussed ina later description of the generation and use of display layouts ofGraphic Icons 144 and other displayable visual objects.

Finally, it must be noted that the functions and operations of VisualLinks Control 164 and Visual Links Organizer 166 are implemented throughthe standard functions and utilities of Microsoft Windows in thepresently preferred embodiment. The functionality and capabilities ofMicrosoft Windows are well known to those in the relevant arts and arewell described in readily available publications, such as “Windows 3.1Programmer's Reference” by James W. McCord and published by QueCorporation and “Inside OLE 2” by Kraig Brockschmidt and published byMicrosoft Press, which are incorporated herein by reference asnecessary. As such, the following will describe Visual Links Control 164and Visual Links Organizer 166 only as necessary to describe andunderstand, for example, the operation of Visual Links Automatic CaptureEngine 138A and of Visual Links Organizer 166 in organizing the displayof Visual Links 142, which will be described in detail.

2. General Description of Visual Links Automatic Capture Engine 138 andVisual Links Organizer 166 (FIGS. 1A-1D, 2A and 2B)

As described above, a primary function of Visual Links Automatic CaptureEngine 138, hereafter referred to generally as a Visual Links AutomaticCapture Engine 138 and representing both Visual Links Automatic CaptureEngine 138A and at least certain aspects of the embodiments of a VisualLink Manual Capture Engine 138M, is to extract graphics information fromthe text and graphics presented on a page of an HTML document, or otherfile containing graphic information, to be incorporated into a VisualLink 142 as a Graphic Icon 144. As will be seen in the followingdescription, and because the icons generated by Visual Link CaptureEngine 138 are formed of graphics information extracted from the HTMLpage or document referred to by the Visual Link 142 containing theGraphic Icon 144, the icons of the present invention provide visuallystrong, distinctive images to be used in visually identifying thecorresponding HTML page or document to a user. In addition, and as willalso be seen from the following, while not in itself a compressed image,a Graphic Icon 144 may require significantly less space to store anddisplay than do the typical compressed “thumbnail” images of the priorart, and less network time to communicate from one system to another.Further, the Graphic Icons 144 of the present image, being generatedfrom the rich variety of images available on HTML documents or pages, orin other graphics files, provide a range and variety of images notgenerally available in the standard types of “icons” used, for example,in Microsoft Windows applications, in part because of the slightlylarger size of Graphic Icons 144 and in part because of the relativeease with which visually distinct Graphic Icons 144 are generated.

An essential function of Visual Link Capture Engine 138 is theextraction of graphics information from the graphically represented textand images present in an HTML document or page, or other form of imagefile, and the forming of that graphics information into a Graphic Icon144.

Visual Link Capture Engine 138 may perform this operation in a number ofways, for example, providing a user with a mechanism to manuallydelineate and copy an area of an image, such as an area of an HTML pageor document or other form of graphic image, to comprise the graphicinformation that is used to create a Graphic Icon 144. Such mechanismsare commonly known and implemented using, for example, the functions andoperations available in the Microsoft Windows operating environment andused, for example, in Microsoft Paint and similar drawing programs. Thisapproach may be used, for example, in an implementation of Visual LinkCapture Engine 138 used in Local System 112, and designated in FIG. 1Aas Visual Links Manual Capture Engine 138M, where a user is creatingVisual Links 142 for a relatively small number of HTML documents orpages or a relatively small number of other image files.

A user may also use the facilities of a graphic paint or annotationsystem to add, replace, or delete, in any combination, or otherwisemodify, selected parts of a selected area. A user may also, as analternative or in combination with such operations, create a part of orthe entirety of the visual image by the graphics operations of such agraphics application program or utility, or by selection of a part of orthe entirety of the visual image from an independent library of images,and may modify an image or a part of an image selected from a library ofimages for such purposes. In such cases, the image part of a Visual Link142 is thereby not derived, at least not totally, from the imagery of anHTML image.

In this regard, it is discussed in the following detailed descriptionsof a Visual Links Automatic Capture Engine 138 that it has been foundpreferable, in the presently preferred embodiment, that the dimensionsand aspect ratios of the Graphic Icons 144 be selected to allowefficient packing into a given display area For these reasons, thedimensions and proportions of Graphic Icons 144 are preferably selectedfrom a relatively small set of possible dimensions and aspect ratios toallow the efficient organization, packing and display of Graphic Icons144 in a directory window. The method for manual selection of an area ofan image to be used in generating a Graphic Icon 144 thereby preferablyrepresents to the user not only the original image, or a section ofinterest thereof, and the area selected by the user for possible use ingenerating the Graphic Icons 144, but also the possible Graphic Icons144 that would result from the proposed selection.

For example, the user may be provided with a mouse controlled “stretchyrectangle” to use in selecting a portion of the original image forpossible use in generating a Graphic Icon 144, in the manner well known,for example, in Microsoft Windows graphics programs such as theMicrosoft Paint program. The Graphic Icons 144 generation program wouldthen display one or more representations of the Graphic Icons 144 thatcould result from the use of the area delineated by the user, therebyallowing the user to select one of the possible Graphic Icons 144 thatwould result from that selection. In another embodiment, and asillustrated in FIG. 1D, the user would use, for example, the mousecursor to designate one comer, such as the upper left comer, of an areaof the original image and the Graphic Icon 144 generation program wouldoverlay and display on the original image a set of rectanglesrepresenting the set of possible Graphic Icons 144 wherein the set ofrectangles conform to the range of dimensions and aspect ratios selectedfor the Graphic Icons 144, the upper left comer of each of the displayedrectangles being located at the initial point designated by the user.The user could then select one of the displayed rectangles asdelineating the area of the original image to be used in generated theGraphic Icon 144, for example, by “clicking” on the lower right comer ofthe chosen rectangle.

As illustrated in FIG. 1D for an exemplary implementation, the upperleft comer of the delineating rectangle, designated in FIG. 1D as comerA, is first positioned so as to identify the upper left of the intendedarea. Thereupon, in the present example, four candidate aspect ratiorectangles are displayed superimposed on the image, the bottom rightcomers being designated in FIG. 1D as comers B, C, D and E. The userthen selects one of the candidate aspect ration rectangles by “clicking”on the bottom right comer of the selected candidate aspect ratiorectangle, whereupon the unselected rectangles cease to be displayed.The user may then re-position the entire selected candidate aspect ratiorectangle to select the most appropriate area of the image by “dragging”the upper left comer of the selected rectangle, and may resize theselected rectangle to fit the most appropriate area by dragging thebottom right comer of the selected rectangle, with the system enforcingthe aspect ratio constraint for the selected rectangle.

Visual Link Capture Engine 138 as implemented in Server System 114,however, will preferably provide an automatic mechanism, designated inFIG. 1A as Visual Links Visual Link Capture Engine 138, for selectingareas of graphics images and forming them into Graphic Icons 144 becausethe large number of Visual Links 142 that must be generated. It may alsobe desirable to provide an implementation of Visual Links Visual LinkCapture Engine 138 for use in Local System 112, however, for example,when a user generally lacks the skill, interest or time to create VisualLinks 142 manually.

Such visual links may, for example and in one possible implementation,be held only temporarily, for example, for the duration of a session,thereby providing a visual reference for revisiting a server that wasaccessed earlier in the session.

In a like manner, it may be preferable to provide an implementation ofVisual Links Manual Capture Engine 138M in Server System 114 for use bythe operators thereof, for example, when a source image or page has nosuitable areas from which Visual Links Visual Link Capture Engine 138can generate a suitable Graphic Icon 144 or when the results generatedby Visual Links Visual Link Capture Engine 138 are ambiguous. Suchresults may occur, for example, when a page or image is entirelycomprised of low contrast visual structures or when all of the visualstructures therein are comprised of fine detail text or graphics.

One solution to this problem is, and for example, to use as the “image”a few words or a legible part of the URL 154, such as the string ofcharacters typically following “www”.

In addition, and if and when the capture engine operates automatically,it may be advantageous to display the document or web page in aselective way, for example, by omitting parts of the page comprisingsmall text or material otherwise not directly and sufficientlydescriptive of the page, such as advertisements. Such parts of a pagemay be omitted by use of the methods described in the following forselecting areas of an image to be used in creating Graphic Icons 144,for example, and advertisements or similar areas of a page may beomitted based upon their non-image contents, such as links to othersites.

A Visual Links Visual Link Capture Engine 138 for selecting areas ofgraphics images and forming them into Graphic Icons 144 must meet anumber of requirements and must provide a number of functions. Forexample, the Visual Link Capture Engine 138 must provide an evaluationprocedure and mechanism for comparing the worth, that is, the visualstrength, of various proposed regions of an image, and at leastaffirming the validity of a region of an image as a source of imageinformation for a Graphic Icon 144. The evaluation procedure willappropriately weight and balance the image areas, bold text areas andlogos, and will be relatively insensitive and unresponsive to areascontaining small fonts and fine details.

The Visual Link Capture Engine 138 will also provide an efficientmechanism for searching an image, such as an HTML page or document orother form of image, and identifying possible areas or regions to beevaluated or selected for use in creating the Graphic Icon 144. In thisregard, a given image may contain an extremely large number of possiblesubregions which may serve as source regions for a Graphic Icon 144; forexample, a 640 pixel by 1480 pixel bitmap image may contain more than 10billion possible regions that might be used to generate a Graphic Icon144.

In addition, the original image may appear as or be comprised of a setof definitions of graphic objects, as may be defined, for example, byhypertext markup language (HTML). In these instances, it may bedesirable to cull the graphics objects for the purpose of omitting fromthe “original” image, and thus from consideration for use in a GraphicIcon 144, such parts of the “original” image as are commonly notsuitable for use in creating a Graphic Icon 144. Examples of such mayinclude fine text, a wallpaper background, or bitmap parts that may withhigh probability be automatically classified as advertisements or otherpictorial subparts irrelevant to the purpose of visually depicting theessential visual appearance of the original image. Again, such parts orcomponents of an original image may be culled based upon the methodsdescribed herein, or upon their contents.

Finally, the image information extracted from a graphics image, whichmay be of various dimensions and proportions, must be shaped into aGraphic Icon 144. In this regard, however, a single fixed aspect ratio,for example, of 3:4, for either the extracted graphics information orthe Graphic Icon 144 is too constraining considering the possible areasor regions that may be selected from a given graphics image as mostdistinctive and most unique to that graphics image, both in itself andrelative to other graphics images. It is preferable, however, toefficiently organize and display the maximum number of Graphic Icons 144in, for example, a directory window, and, for this reason, thedimensions and aspect ratios of the Graphic Icons 144 will preferably beselected to allow efficient packing of any arbitrary set of icons into agiven display area. For these reasons, the dimensions and proportions ofGraphic Icons 144 are preferably selected from a relatively small set ofpossible dimensions and aspect ratios to allow the efficientorganization, packing and display of Graphic Icons 144 in a directorywindow.

Visual Link Capture Engine 138 will therefore provide an Visual LinkCapture Engine 138 that automatically shapes the extracted imageinformation into a Graphic Icon 144 selected from a relatively small setof possible dimensions and aspect ratios that are selected for efficientpacking into a display area

In a related problem, Visual Links Organizer 166 will provide amechanism for efficient packing any arbitrary collection of GraphicIcons 144 having various dimensions and aspect ratios from that set ofdimensions and aspect ratios into a display area

First considering the evaluation procedure and mechanism for comparingthe worth, that is, the visual distinctiveness, of various proposedregions of an image, and affirming the validity of a region of an imageas a source of image information for a Graphic Icon 144, it is apparentthat bold text and pictorial material, that is, large or highlycontrasting features or areas of text or image, are significantly morevisible and distinctive than are areas of, for example, fine text or lowcontrast pictorial material.

According to the present invention, as and described in further detailbelow, the most visually significant features of an image, that is, thelargest and highest contrast or highest visibility features, areemphasized, and the less significant features suppressed, by coarseningand blurring the overall image, which is usually in the form of a bitmapimage. Also according to the present invention, the features or regionsof most significance in an image will generally be as evident in agray-scale version of a bitmap as in a color version, so that agray-scale version of the image may be used rather than the colorversion to simplify and speed the task of identifying the regions orfeatures of most significance.

Pursuant to evaluation, therefore, a coarse grayscale version of theoriginal image, for example, ¼ the linear size in pixels of the originalimage, is low-pass filtered, and an “edge” image is made from the coarsegrayscale version of the original image, wherein the edge image isregions having high numerical values in the regions of highnon-uniformity of the filtered image. The amount of blurry “edge” in anysubrectangle, relative to its size, is taken as an approximation of thedegree to which this rectangular area contains visually bold orsignificant information.

It is anticipated that some very small regions of the coarse grayscaleversion of the original image may contain high evaluations ofedge-per-area, yet not be suitable for use as Graphic Icons 144.Therefore, and according to a presently preferred embodiment of theinvention, candidate areas are required to be of predetermined minimumsize and the criteria by which regions are rejected for considerationincludes size. As such, and in the present implementation, if two areasexhibit the same average amount of edge-per-pixel, then the larger ofthe two is deemed to be the better region to be used in creating theGraphic Icon 144.

Because the Visual Link Capture Engine 138 searches for bold and coarsefeatures, and because, for reasons discussed in the following, it is notnecessary to locate the boundaries of the Graphic Icon 144's source areain the original image or in the coarse grayscale version of the originalimage with pixel accuracy, the Visual Link Capture Engine 138 performs afirst search with a second, very low-resolution image. The very lowresolution image is constructed, for example, made from the ¼ size edgeimage and is, for example, {fraction (1/16)} the linear size of theoriginal image, or ¼ the size of the ¼ scale edge image. The use of sucha low resolution image, have a proportionally smaller number of totalpixels and a correspondingly smaller number of pixels in each sub-regionthereof, the Visual Link Capture Engine 138 can perform a relativelythorough search, involving crude test-for-appropriateness of a very widerange of sizes, locations and aspect ratios of initially selectedregions of the original image. Subsequent refinement of the area orareas thus found is then based on the more accurate ¼ size edge image.

Once a suitable area of the ¼ size edge image is found, the candidatearea, which in the present embodiment of the invention is a rectangle,is refined by the process of iteratively pushing one or another side ofthe rectangle outward by one row of pixels so long as this increases therectangle's worthiness for use in creating the Graphic Icon 144, andfinally adjusting each side of the rectangle inward or outwardaccordingly as the rows of pixels just inside or just outside of therectangle have or do not have significant edge indications. That is, aside of the rectangle is retracted the if the strip of pixels insidethat side contains little or no edge feature and the side of therectangle is extended if the edge indications in the strip of pixelsoutside that side indicate the continuation of features already partlycontained in the rectangle.

Finally, it has been described above that the rectangle of imageinformation extracted from a graphics image as representing the visuallymost distinctive and unique area of the original image may be of variousdimensions and proportions. It has also been described that the selectedrectangle of graphic information is then preferably shaped into aGraphic Icon 144 having a size and aspect ratio selected from arelatively small set of possible dimensions and aspect ratios to allowthe efficient organization, packing and display of Graphic Icons 144 ina directory window.

In a presently preferred embodiment of the invention, the set ofpossible aspect ratios for the final Graphic Icons 144 have been chosento include aspect ratios of 6×2 (units of height to width), 4×3, 3×4 and2×6 as these aspect ratios result in Graphic Icons 144 having the sametotal area and because an arbitrary group of Graphic Icons 144 havingthese aspect ratios will pack efficiently into a given display area. Theultimate icons will therefore be resized, according to one exemplary setof sizes, to be 120 pixels high by 40 pixels wide, or 80 pixels high by60 pixels wide, or 60 pixels high by 80 pixels wide, or 40 pixels highby 120 pixels wide, or they may be slightly less in one or bothdimensions but will be deemed nominally to “occupy” these screen displayareas. Accordingly, and because the region of the original image thathas been selected as visually most distinctive and unique will rarelyhave one of these predetermined aspect ratios, a final adjustment of thearea or region of graphic information that has been selected andextracted from the original image is necessary to conform the selectedarea or region to one of these predetermined aspect ratios.

A number of such adjustment processes are possible, and the presentlypreferred embodiment of the Visual Link Capture Engine 138 may or doesuse any or any combination of these adjustment processes. For example,in a first process, the icon may be left in the shape as extracted fromthe original image and placed in a final display into the best-fittingshape of area, leaving blank unused area on left and right, or on topand bottom. In a second process, the icon can be distorted by a verticalor horizontal stretch so that it fits such an area and, in a thirdprocess, the area from which the icon is cut may be extended eithervertically or horizontally to match an acceptable shape while, in afourth process, a combination of stretching, of up to, perhaps, 10%, maybe combined with an extension of the source area. Finally, in yetanother method, and if and where a background color can automatically bediscerned, that color may be used to fill the extra area so that theicon's sides do not make a disconcerting cut through other visualobjects in the vicinity.

In all cases, therefore, the Graphic Icons 144 are produced byappropriate scaling, so that they each occupy the same amount of screenarea and pack efficiently into the display screen area, such asdirectory window. In addition, and as discussed in the following, theGraphic Icons 144 may be but are not necessarily all created to the samecolor scheme, such as the 256 color standard used in Microsoft Windows,so that they can be displayed simultaneously, and in certainimplementations the final color compatibility may be achieved by a colorconversion just preceding the display of the icons to a user. Finally,and in addition to the above described advantages, each display ofGraphic Icons 144, such as a directory or folder in the directory, will,as a whole, present a certain distinctiveness because of its near-uniquecombination and placement pattern of Graphic Icons 144, therebyproviding the user with a further visual aid for looking for,remembering and recognizing individual Graphic Icons 144.

Finally, the various images and versions of images used by the VisualLink Capture Engine 138 of the present invention and described above areillustrated in FIG. 2A wherein Image 210 represents the original imageof an HTML page or document, or other graphics image, and GrayscaleImage 212 represents the ¼ size grayscale image generated from OriginalImage 210 and Filtered Grayscale Image 214 represents low pass filtered,that is, coarsened and blurred image generated from Grayscale Image 212and Edge Image 216, in turn, represents the edge image generated fromFiltered Grayscale Image 214. Coarse Edge Image 218 then represents the{fraction (1/16)} scaled image generated from Edge Image 216 forpurposes of identifying candidate regions from which to construct theGraphic Icon 144, while Graphic 44 Image 220 represents the region ofOriginal Image 210 that has been selected, as described above, forconstruction of the Graphic Icon 144, and Graphic 44 represents Graphic44 Image 220 after adjustment to one of the predetermined sizes andaspect ratios chosen for Graphic Icons 144, that is, Graphic 44 is thefinal Graphic Icon 144.

It will be noted that the presently preferred embodiment of Visual LinkCapture Engine 138 operates with grayscale images, that is, the relative“brightness” of image components, in identifying those areas of anoriginal image most suitable for incorporation into a Graphic Icon 144.As such, the following descriptions of Visual Link Capture Engine 138will primarily focus on the grayscale implementation of the processesfor generating Graphic Icons 144.

Alternate implementations of the Visual Link Capture Engine 138,however, may utilize the color information, that is, such as the red,green and blue components, of Original Image 210 in identifying thevisually most significant and unique region of the image. The methods,processes and mechanisms of the capture engine of the present inventionas implemented for the use of color information are very similar tothose for the use of grayscale images, however. As such, the use ofcolor image information will be discussed briefly in the following,using FIG. 2A for purposes of illustration, before continuing with adetailed discussion of the mechanisms using grayscale information andthose of ordinary skill in the relevant arts will understand theadaptation of the grayscale mechanisms to the use of color imformation.

In a presently preferred embodiment of a Visual Link Capture Engine 138using color information, the colors are not treated separately throughthe entire process. Instead the color information is included orcombined into a single edge image corresponding to and equivalent to ¼scale Filtered Grayscale Image 214 and this color Filtered GrayscaleImage 214 is then used in the processes and mechanisms described in thefollowing in the same manner as a grayscale Filtered Grayscale Image214.

As will be described in the following with regard to the grayscaleimplementation of Visual Link Capture Engine 138, the amount of “edge”at a given pixel of an image is an indication of how much the“brightness” of the image is changing at that pixel and is thereby anindication of the distinctiveness of that region of the image relativeto other regions of the image. In the implementation of Visual LinkCapture Engine 138 using color information, however, the amount of“edge” at a pixel of the image is an indication of how much the red orgreen or blue components of the image, or any combination of the red,green or blue components of the image, are changing in that region, aswell as an indication of how much the “brightness” of the image ischanging at that region. The amount of “edge” therefore reflects theamount of “color-and-brightness” change, which is generally referred toin brief as the “color” change.

This embodiment of Visual Link Capture Engine 138 thereby determines a“color change” at each pixel of the blurred image wherein the colorchange may be usefully defined and used as any of a number of measures.For example, the color change may be defined and determined as the redcomponent change plus the green component change plus the blue componentchange plus the brightness component change at a pixel, wherein thebrightness component change is the only component considered in thepresently preferred grayscale implementation of the capture enginedescribed below. In one implementation, the red component change at apixel may, for example, be determined as the red component left-rightchange across the pixel plus the red component above-below change acrossthe pixel, and likwise for the green, blue and brightness components.The left-right and above-below changes at a given pixel and for a givencomponent may, in turn and for example, be defined as the absolutevalues of the differences between that component at the pixels to,respectively, the left and right and above and below that pixel. In yetother implementations, the change in a component may be defined aseither the larger of the absolute value of the difference in thecomponent between the top and bottom neighbors of each pixel or theabsolute value of the difference in the component between the right andleft neighbors of each pixel, thus separately considering changesoccurring in the horizontal and vertical directions.

The “amount of edge” at given pixel, in turn, may be determined as theaverage, for that pixel, of each of the component changes and, inanother implementation, as the maximum of the component changes, ratherthan the average.

Whichever method is used, this process thereby determines for, eachpixel, a value depending upon both how much the brightness component ischanging in a given direction at each pixel and how much the red, greenand blue components are changing in that direction at each pixel. Forexample, given a “pie chart” wherein the different sections of the charthave approximately the same brightness but are drawn in differentcolors, this process will determine significant “edge” along the radialdivision lines between the sections of the chart.

It will be noted that in yet another alternate implementation of thecapture engine using color information, Visual Link Capture Engine 138may generate three Grayscale Images 212 wherein each Grayscale Image212, respectively designated in FIG. 2A as Red Color Image 212R, GreenColor Image 212G, and Blue Color Image 212B, contains the grayscaleequivalent values of, respectively, the red, green and blue componentsof Original Image 210. The Visual Link Capture Engine 138 will thenexecute the same operations as described above, but for each of thethree Color Images 212R, 212G and 212B separately, and will generatethree corresponding Coarse Edge Images 218, one corresponding to each ofRed Color Image 212R, Green Color Image 212G, and Blue Color Image 212B.In one implementation, the three Coarse Edge Images 218, which couldpossibly correspond to three different candidate regions of OriginalImage 210, are then evaluated as described above to select the visuallymost significant and unique region of the original image to use inconstructing the Graphic Icon 144 as described above.

In one alternate implementation of the three color image process, theVisual Link Capture Engine 138 uses the three original Coarse EdgeImages 218 corresponding to Red Color Image 212R, Green Color Image212G, and Blue Color Image 212B to select the region or regions of theoriginal image providing the visually most significant and unique regionor regions of the original image, each being selected on the basis of asingle color component, that is, red, green or blue. The result mayagain be one, two or three different candidate regions of the originalimage, depending, for example, on whether any of the single colorcomponent Coarse Edge Images 218 overlap. The Visual Link Capture Engine138 then generates a Red Color Image 212R, Green Color Image 212G, andBlue Color Image 212B for each candidate region selected on the basis ofthe individual color components. In most instances, however, theautomatic color mechanism need only generate two new grayscale imagesfor each region as each region will already be represented by onegrayscale image. For example, if a given region has been selected ofcontaining the visually most significant area of Original Image 210based upon the red color component, a Red Color Image 212R and thecorresponding Filtered Grayscale Image 214, Edge Image 216 and CoarseEdge Image 218 will already exist, so that the Visual Link CaptureEngine 138 need generate this images only for the blue and greencomponents.

The capture mechanism would then perform the above analysis of eachcandidate region, but three times, once for each color component of eachcandidate region, and combines the three evaluations for each candidateregion to generate a value representing the visual merits of eachcandidate region. The Visual Link Capture Engine 138 then selects thevisually most significant of the candidate regions to use inconstructing the Graphic Icon 144 as described above.

Returning to consideration of the operation of Visual Link CaptureEngine 138 using only brightness information, the above describedcapture of image information and creation of a Graphic Icon 144 can beillustrated by way of an example. In this instance, it is assumed thatthe Original Image 210 exists in one of the more complex image formats,such as the 256-color bitmap image format wherein each byte value 0-255corresponds to a color whose red, green and blue components are definedin an auxiliary color lookup table (LUT).

The Visual Link Capture Engine 138 first reads the color LUT andcomposes a gray-scale LUT whereby the brightness of each of the 256colors is taken to be the average of that color's red, green, and bluecomponent values. It will be noted in this regard that alternative andmore refined measures of brightness are known to those of skill in therelevant arts, but this method is sufficient for the present embodimentof the Visual Link Capture Engine 138. Then, and assuming for purposesof the present example that the bitmap comprising Original Image 210 isfree of border, marginalia or other extraneous accouterments, theOriginal Image 210 bitmap is read and processed into a ¼ size grayscaleimage comprising Grayscale Image 212. As such, 116 source pixels ofOriginal Image 210 contribute to each destination pixel of GrayscaleImage 212 and this conversion is accomplished by adding and averagingthe brightness of the 116 source pixels corresponding to each GrayscaleImage 212 pixel, as taken from the gray-scale LUT.

The low-pass filtered image comprising Filtered Grayscale Image 214 isthen created from Grayscale Image 212 by use of a kernel array ofcoefficients, that is, by multiplying neighboring pixels of GrayscaleImage 212 by the numbers of the kernel array and dividing by 18, whereinan exemplary kernel array may be represented by

0 0 1 0 0 0 1 2 1 0 1 2 2 2 1 0 1 2 1 0 0 0 1 0 0,

and wherein the value 18 is the sum of the multipliers in the kernel.

In then creating Edge Image 216, it must be noted that the value of eachpixel of Edge Image 216 depends on the values of the left, right, topand bottom neighbors of the corresponding source pixel in FilteredGrayscale Image 214. The value of each pixel of Edge Image 216 maytherefore be found by finding the brightness difference between left andright neighbors and the brightness difference between top and bottomneighbors, and then adding the absolute values of these values,multiplying each result by a first constant, subtracting a secondconstant, and clipping the result for each pixel of Edge Image 216 to aminimum value of 0 and maximum value of 255. Alternatively, the value ofeach pixel of Edge Image 216 may be determined as the absolutedifference between the corresponding left and right vs. top and bottompixels of Filtered Grayscale Image 214, which tends to emphasizehorizontal and vertical edges in the images. In yet another alternateimplementation, the same method may be executed, but using the red,green and blue color components, as described above, thereby causing theVisual Link Capture Engine 138 to evaluate the candidate regions of theoriginal image on the basis of image structures exhibited by differencesin color as well as brightness.

The value assigned to each block, or coarse pixel, of the resultingextremely coarse {fraction (1/16)} size edge image, that is, Coarse EdgeImage 218, is then the sum of a corresponding 4×4 pixel block of EdgeImage 216 pixels, these values being stored as word-size values ratherthan byte-size values.

The Visual Link Capture Engine 138 then determines the rectangularregion to be used as source for the Graphic Icon 144 by executing fourdistinct steps, or operations:

(1) an initial rectangular region is found by exhaustive search in thevery coarse image, Coarse Edge Image 218;

(2) by reference to the ¼ size image, Edge Image 216, this area isextended on any or all sides so long as its figure of merit increases;

(3) by iterative successive treatment, each side of the grown rectangleis adjusted either in or out as best fits the edge structure found inthe local area (cutting back if the rectangle has extended into emptyspace and pushing out if structure seems to have been truncated); and,

(4) the otherwise “final” area is adjusted to conform to one of theselected aspect ratios to form the Graphic Icon 144.

Considering each of these operations further, the mechanism forexamining candidate rectangles will be discussed in a following detaileddescription and it will shown therein that the mechanism used in thepresently preferred embodiment of Visual Link Capture Engine 138 permitsfast summing values in candidate rectangles, thereby allowing VisualLink Capture Engine 138 to test a large number of candidate rectanglesquickly and efficiently. As such, the initial candidate rectangle isfound by exhaustive examination of all candidate rectangles in a CoarseEdge Image 218 having widths in the range of 6 through 10 pixelsless-than-full-width pixels and heights in the range of 6 through 10less-than-full-height pixels. That is, the starting rectangle must be atleast 6 pixels of the {fraction (1/16)} size image and at most ten lessthan the full size rectangle in both the horizontal and verticaldirections so that the search in the coarse space is exhaustive exepctfor very thin and very tall or very wide rectangles. The evaluation ofthese candidate rectangles on the same basis that is subsequently usedto evaluate growth of the selected rectangle, which will thereby beequally exhaustive and with the same possible exceptions. It will alsobe noted that 6-through-10 pixel high by 6-through-10 pixel widerectangles of Coarse Edge Image 218 were selected as providing areashaving an acceptable probability of containing at least parts ofvisually significant image structures while being sufficiently small asto reduce the processing burden.

As described above, the search for and subsequent modification ofsuitable candidate rectangles for the generation of Graphic Icons 144requires a method for the evaluation of each rectangle or modificationthereof, that is, the determination of a “figure of merit” or “figure ofassurance” for each candidate rectangle. The search for a suitable areaof an image to comprise of Graphic Icon 144, or a proposed modificationof a candidate rectangle to determine whether the candidate rectangle isimproved by moving the sides of the rectangle inwards or outwards,results in each case in a local maximum in the total search spacewherein the local maximum has a “value” and a “sharpness” which togethercan be used to express a figure or merit or assurance that the rectangleis the or a proper area to be used in generating a Graphic Icon 144. Inaddition to being used by Visual Link Capture Engine 138 to evaluate acandidate rectangle or a modification to a candidate rectangle, thefigure of merit or assurance can then be used to flag or otherwiseindicate results having a low figure of assurance for subsequent humanapproval or correction, for example, by use of a Visual Links ManualCapture Engine 138M.

It will be appreciate by those of ordinary skill in the arts that thevalue component of the figure of merit or assurance of a candidaterectangle is a function of the location of the candidate rectangle inthe image and the size of the candidate rectangle. That is, the locationof a candidate rectangle determines which area of the image is includedin the rectangle and, in turn, is a function of the average amount ofimage “edge” in the candidate rectangle, wherein “edge” may be definedas the local degree of visual coarse structure non-uniformity of imagein the selected area of the image. The degree of local visual coarsestructure non-uniformity, in turn, may be generally defined as theamount of contrast, or difference, in brightness or color betweenadjacent areas in the rectangle and the “sharpness”, or abruptness, ofthe boundary between adjacent areas of different brightness or color inthe rectangle, that is, how fast does the change in brightness or coloroccur when moving from one area to another.

The criteria of “sharpness” also applies in the steps wherein the sidesof a candidate are adjusted inwards or outwards to incorporate theentirety of a visually strong local structure already at least partiallyincluded in the candidate rectangle into the final candidate rectangle.In this sense, “sharpness” is a measure of how fast and in whatdirection the figure of merit or assurance value of the candidaterectangle changes per unit distance as a side of the rectangle is movedinwards or outwards.

In this regard, it will be appreciated by those of ordinary skill in thearts with respect to the determination of a figure of merit or assuranceof a candidate rectangle that the step of generating Filtered GrayscaleImage 214 is essentially a pre-processing step with respect to thesearch for and evaluation of candidate rectangle wherein small text andlow resolution or low contrast areas of the original image are discardedas being of little value in generating a Graphic Icon 144 presentingstrong visual components to a viewer.

Finally, the size of the rectangle determines the amount of originalimage incorporated into the candidate rectangle and in the presentlypreferred embodiment of Visual Link Capture Engine 138 is selected onthe general principle that a larger candidate rectangle is preferableover a smaller candidate rectangle.

In the presently preferred embodiment of Visual Link Capture Engine 138,therefore, the figure of merit or assurance is determined as a functionof the average amount of “edge” per pixel in the rectangle and adding acontribution value that increases, up to a limit, with increasing sizeof the rectangle. In the presently preferred embodiment, the actual sizecontribution has two breakpoints with increasing value of the perimeterof the rectangle, such that for rectangles beyond a certain size thereis no further value added for a further increase in size of therectangle. This reduces, but does not eliminate, the possibility of verysmall disproportionally structured areas from becoming Graphic Icons144, and prevents a potential “run-away” condition whereby the “figureof merit” increases with the size of the rectangle regardless ofcontent.

The near-final operation of pushing a side of a rectangle in or out isperformed by computing the amount of edge in the strip of pixels justinside the rectangle and, if significantly lower than average, excludingthe strip from the rectangle, that is, pulling the side of the rectangleback, or, if the strip just outside the side of the rectangle contains asignificant amount of edge structure, indicating that image structurehas been truncated that probably should be included in the region,including the strip in the rectangle, that is, pushing out that side ofthe rectangle.

Finally, the final sizing and shaping of the Graphic Icon 144 area hasbeen discussed above. However, for the case in which no “filling” or“stretching” is to be done, but the source area is made to conform to anacceptable ratio of, say 6:2 or 4:3 of 3:4 or 2:6, the following are thesteps for deciding which sides are to be pushed out, relative to theexisting rectangle.

These “rules” for the steps for final sizing and shaping may berepresented schematically, including the additional in-between ratios of5:9 and 1:1 and 9:5 as division points, by:

→6:2←9:5→4:3←1:1→3:4←5:9→2:6←

The meaning of the schematic is:

if the height:width ratio is greater than 6:2, widen the area to 6:2

else if the ratio is greater than 9:5, stretch its height to 6:2;

else if it is greater than 4:3, widen it to make it 4:3;

else if it is greater than 1:1, make it taller so as to make it 4:3;

else if it is greater than 3:4, widen it to make it 3:4;

else if it is greater than 5:9, make it taller to make it 3:4;

else if it is greater than 2:6, widen it to 2:6;

else make it taller to make it 2:6

It should be noted that in the example described above, the finaladjustment for a selected aspect ratio has, for the sake ofillustration, been slightly simplified. For example, in the most commonimplementation the final Graphic Icon 144 will have a thin border areaon all sides for the purpose of delineating it from neighboring icons inscreen layouts containing many icons, as illustrated in FIG. 2B. Thecandidate areas are therefore slightly smaller, and ratios slightlydifferent, if no distortion whatever is to be allowed in re-sizing fromthe source image to the icon, although a slight re-sizing may not onlybe acceptable but even desirable in order to use a more desirably shapedor sized region from the source image.

It must also be noted that the “tiling” of many icons of the variousacceptable sizes and shapes, as illustrated in FIG. 2B, is a“bin-packing” problem of tiling geometry and, in certain implementation,such as with a relatively small number of icons per display, forexample, 10 to 20, Visual Links Organizer 166, which will be discussedfurther in the following descriptions, may be implemented by computingand tabulating one arrangement for each maximally achievable iconpopulation distribution. Thus, the possible arrangements for a displayor a sub-section of a display can be determined in advance, aspre-arranged packing patterns for packing, for example, two 6:2s, four4:3s, three 3:4s and one 2:6, and stored for use in a display or asub-section of a display as necessary.

Finally, it must be noted that the Original Images 210 from whichGraphic Icons 144 are generated will generally be acquired, or captured,by Local System 112 or Server System 114 as a graphics file or componentof a standard type, such as a GIF or JPEG image or an HTML document orpage containing graphic information in the form of GIF or JPEG images,and that the graphic file or component will most usually be a colorimage in any of several different color information formats. There aremany well known and available methods and utilities for convertingbetween GIF and JPEG images and, while it is preferable that Visual LinkCapture Engine 138, Visual Links Control 164 and Visual Links Organizer166 operate with a standard image format, the conversion between oneformat and another is well understood by those of ordinary skill in therelevant arts and will not be discussed further herein.

As regards color information formats, it is again preferable that VisualLink Capture Engine 138, Visual Links Control 164 and Visual LinksOrganizer 166 use a standard color format, of which there are several.For example, the 256 color format as used in Microsoft Windows, provides256 numeric values that may be assigned to corresponding color valueswherein the color of each pixel is coded in one byte of information as aRGB (red, green, blue) value and wherein each of the 256 differentnumeric values may be assigned to a different combination of red, greenand blue and thus to a different color. In the “high color” format, eachpixel's red, green and blue values are stored in 5 or 6 bit fields of a16 bit word, and in the “true color” format each pixel's red, green andblue values are expressed directly in three corresponding bytes, giving256 different possible values each of red, green and blue.

It is also preferable that the color format chosen for use in VisualLink Capture Engine 138, Visual Links Control 164 and Visual LinksOrganizer 166 be compatible with, for example, Microsoft Windows andMicrosoft Windows compatible application programs, such as Netscape andMicrosoft Explorer, as it is anticipated that Visual Links 142 will beused and displayed in such an operating environment and with suchapplication programs. For this reason, the presently preferredimplementations of Visual Link Capture Engine 138, Visual Links Control164 and Visual Links Organizer 166 generate, store and display GraphicIcons 144 in a 256 color format that is compatible with MicrosoftWindows and Microsoft Windows compatible application programs, such asNetscape and Microsoft Explorer.

In this regard, it should be noted that the 256 color format allows eachof the 256 different numeric values that can be expressed in a singlebyte to be assigned arbitrarily to a different color and that, as aconsequence, the 256 colors used in Visual Link Capture Engine 138,Visual Links Control 164 and Visual Links Organizer 166 need not beidentical to those assigned to corresponding numeric values in MicrosoftWindows and Microsoft Windows compatible application programs, such asNetscape and Microsoft Explorer, so long as the color assignments aresufficiently similar to be visually compatible.

It is possible, however, that a given Original Image 210 will use a 256color format in which the color/numeric value assignments are visuallyincompatible with the assignments used in Visual Link Capture Engine138, Visual Links Control 164 and Visual Links Organizer 166 or inMicrosoft Windows and, in these cases, the reassignment of color/numericvalues is easily accomplished through a lookup table, as well known inthe relevant arts. In a like manner, an Original Image 210 in the highcolor or true color formats may be readily converted into the chosen 256color format by determining the bit and byte assignments to red, greenand blue value, unpacking the color information from the format intored, green and blue values, and converting the red, green and bluenumeric values into comparable red, green and blue values in the chosen256 color format, either through lookup tables or, in some instances, bynumeric calculation.

Finally in this regard, it should be noted that Visual Link CaptureEngine 138 will perform a conversion of the Original Image 210 from thereceived color format to the chosen 256 color format before theGrayscale Image 212 is generated from the Original Image 210, so thatthe generation of Grayscale Images 212, and all subsequent operations,including the final generation of the Graphic Icon 144, are performed ina standard manner. As illustrated in FIG. 2A, it should also be notedthat it may be desirable to display an image of the Graphic Icon 144 orpotential Graphic Icons 144 during operation of Visual Link CaptureEngine 138, such as in the implementation of Visual Link Capture Engine138 used in Local Server 112 where the user is actively interacting withthe system. In these applications the presently preferred implementationof Visual Link Capture Engine 138 generates and displays a temporaryrepresentation of the Graphic Icon 144 or potential Graphic Icons 144being considered or generated to the user using the original colorformat of the Original Image 210, thereby avoiding the need to perform acolor format conversion for a temporary display. This image orrepresentation of the Graphic Icon 144 is not stored or saved, however,and only the 256 color format final Graphic Icon 144 is stored for lateruse.

Lastly, it has been described above that certain implementations ofVisual Links Capture Engine 138, such as Visual Links Manual CaptureEngine 138M in Local System 112, provide a user with the capability tomanually designate a region of Original Image 210 and to convert thedesignated region of Original Image 210 into a Graphic Icon 144. Theseimplementations of Visual Link Capture Engine 138 preferably utilize thegraphic finctions and capabilities provided in the operating environmentin which Visual Link Capture Engine 138, Visual Links Control 164 andVisual Links Organizer 166 are executing, such as Microsoft Windows. Inthese implementations, the Visual Links Manual Capture Engine 138M andVisual Links Control 164 provide interfaces to, for example, theMicrosoft Windows graphics functions which allow the user to delineateand copy a selectable rectangular region of an image, as well known andunderstood by those of ordinary skill in the relevant arts.

The selected region is then provided directly to the Visual Link CaptureEngine 138 final shaping and sizing functions, described above anddescribed further below, which perform the final steps of generating aGraphic Icon 144 from the selected region. It should be noted that, incertain implementations, it may be desirable to limit the user'sselection as regards the size and proportions, or aspect ratios, of theselected region, as described herein above.

3. Detailed Description of Visual Link Capture Engine 138

Having described the overall operation of a Visual Link Capture Engine138, the following will describe an implementation of Visual LinkCapture Engine 138 in greater detail. It will be noted, in this regard,that certain aspects of the structure and operation of Visual LinkCapture Engine 138, such as the methods and mechanisms for convertingbetween color formats, generating Grayscale Images 212 from OriginalImages 210, Filtered Grayscale Images 214 from Grayscale Images 212,Edge Images 216 from Filtered Grayscale Images 214, and Coarse EdgeImages 218 from Edge Images 216, will be understood by those of ordinaryskill in the relevant arts from the above discussions of an Visual LinkCapture Engine 138 and will accordingly not be described in furtherdetail except as would assist in understanding the present invention.

a. General Description of Visual Link Capture Engine 138 (FIGS. 3A-3H)

Referring now to FIG. 3A1 and 3A-2, the Visual Link Capture Engine 138is implemented in a presently preferred embodiment in Local System 112and in Server System 114 as a plurality of program modules residing inDisk Drive 156 or Memory 158 and controlling the operations of Processor160 to execute the operations described herein on image data residing inDisk Drive 156 or Memory 158. As is well known and understood in therelevant arts, such programs control the operations general purposeProcessor 160 and its associated components to operate as one or morededicated, single purpose mechanisms for performing specific operationsthat are determined by the programs. Finally, it is assumed for purposesof the present discussions that Server System 210 and Local System 212are operating under the control of and incorporate the functions andcapabilities of a well known and understood operating system such asMicrosoft Windows or the general equivalent providing object basedprocessing, object linking, and a variety of functions and operationsthat are generally available to the program creator, such as graphicsediting functions.

The organization of Visual Link Capture Engine 138 as a group ofprocessor control program modules and the functional mechanisms providedby each of the program control modules is illustrated in FIG. 3A and inthe exemplary program listings provided in Appendix A.

Before beginning the following detailed description, however, certainaspects of the control program listings provided in Appendix A should beunderstood, as should certain technical details and assumptions of thefollowing descriptions. First, the control program listings providedinclude certain versions of the program modules having the extension“.noc”. These listings contain alternate implementations of certainprogram modules that may be used in alternate embodiments of the presentinvention, and implementations that, for example, provide displays thatare not necessary for essential operation of the present invention butthat provide displays that may be advantageous for a user or for one whomaintains the programs.

For example, there may be implementations in which immediate visualfeedback to a user is desired, wherein the Visual Link Capture Engine138 operates first to select a potential Graphic Icon 144 region ofOriginal Image 210 and presents the selection as a suggestion that theuser may often accept without revision. In other instances the user maychoose to intervene and, taking some time and care, adjust the sourceregion from Original Image 210 or pick an entirely different sourceregion from Original Image 210. Other alternate embodiments illustratedin these listings utilize red, green blue color information in selectionof regions of Original Image 210 to convert to a Graphic Icon 144, againas discussed above.

In addition, the following descriptions assume that Original Images 210are in the form of “.bmp format” screen captures of a 640-pixel-wide by480-line-high screen, used in color-mapped mode with 256 colors. In thefollowing examples, only 408 of the 480 lines are actually used, and thecolormap is assumed to start in byte [54] of the file, each entryconsisting of the four bytes: blue value, green value, red value, zero.It is also assumed that, immediately thereafter (starting in byte[1078]), is the byte-per-pixel data, appearing bottom line first. It isfurther assumed for purposes of the following discussions that suchfiles, or RAM bitmap equivalents, will be available without marginalframes, Windows buttons, sliders, etc. In the exemplary embodimentpresented herein, such marginalia are stripped by strictly ad hoc meanswherein for each image there is side information describing how thepicture is to be stripped—by replacing a certain amount of the sidematerial with additional white or black background. It will beunderstood by those or ordinary skill in the relevant arts that theseand other techniques may be used to obtain Original Images 210, and howthe system described herein may be readily adapted to other imageformats and to other technical assumptions.

As illustrated in FIG. 3A, the control programs comprising VisualCapture Engine 138 include a Main Program 310 which feeds informationabout images, one at a time, to an image processing mechanism comprisedof a Processing Program 312 and a Find Area Program 314. As will bedescribed in further detail below, Processing Program 312 generatesGrayscale Images 212 from Original Images 210 and passes GrayscaleImages 212 to Find Area Program 314, which generates Filtered GrayscaleImages 214, Edge Images 216 and Coarse Edge Images 218 and identifiesappropriate areas of Original Image 210 to iconize, that is, to convertinto Graphic Icons 144, by searching for locations of bold image, thatis, areas of Original Image 210 holding large fonts, photos, logos orcombinations of these.

b. Processing Program 312—Grayscale Image Generator 316

For the reasons stated above, and for reasons of speed, the searchperformed by Find Area Program 314 is based on one or more smallversions of Original Image 210 and, in the present implementation, thesearch is performed using a ¼ size gray-scale images, that is, GrayscaleImages 212, generated by a Grayscale Image Generator 316 implemented byProcessing Program 312, and derivatives of Grayscale Image 212, that is,Filtered Grayscale Image 214, Edge Image 216 and Coarse Edge Image 218,generated by Find Area Program 314.

In alternate embodiments, however, the search process may be performedwith images having less or more reductions, depending upon theprocessing power and speed of Local System 112 or Server System 214 andthe time available to process each image. It should be noted, in thisregard, that the loss of quality of result appears not be significantfor greater reductions in image size as the object of the search is onlyto identify regions of strong, bold image. Larger images are desirable,however, at least in those implementations providing the user with adisplay of the proposed Graphic Icon 144 areas under consideration, asproviding a more legible and detailed image to the user, and themodifications to Processing Program 312, and to Find Area Program 314,to generate and use images having greater or lesser reductions will bewell understood by those of ordinary skill in the relevant arts.

It should also be noted, as discussed above, that Processing Program 312and Find Area Program 314 may be implemented to use the red, green andblue color components of Original Image 210, as well as the brightnesscomponent, in searching for and identifying the areas of bold imagesuitable for Graphic Icons 144 and, in this implementation, ProcessingProgram 312 may generate, for example, ¼ scale grayscale images of thered, green and blue components of Original Image 210 for use by FindArea Program 314, and for display to the user.

It has been found, however, that acceptable results are normallyobtained from brightness information alone. As a consequence, in thepresently preferred embodiment, Processing Program 312 transforms theRGB (red, green, blue) Original Image 210 into a grayscale image, asdiscussed previously, by using a Color Lookup Table 318A to compose aGreyscale Lookup Table 318B which is used, for each pixel in an OriginalImage 210, to determine greyscale equivalent values for the red, greenand blue values associated with each of the 256 numbers. A GrayscaleConverter 320, in turn, receives these values and sums these values for4×4 blocks of Original Image 210 pixels to generate a “Brightness” Value322 for each 4×4 block of Original Image 210 pixels, the resultingBrightness Value 322 for each block becoming the grayscale value for acorresponding single pixel of the corresponding ¼ scale Grayscale Image312. It will be appreciated by those of ordinary skill in the arts thatGrayscale Converter 320 may generate Grayscale Images 212 at sizes otherthan ¼ scale by corresponding changes in the numbers and arrangements ofthe Original Image 210 pixels selected to form each corresponding pixelof the Grayscale Image 212.

The color component images, that is, Grayscale Images 212R, 212G and212B are constructed in an analogous manner, but wherein each isgenerated using only the values for a single corresponding colorcomponent, that is, red, green or blue.

c. Mechanisms of Find Area Program 314, Introduction

Once a ¼-size grayscale image Grayscale Image 212 is available fromProcessing Program 312, Find Area Program 314 provides the mechanismsfor performing the operations necessary to generate the remaining imagesused in finding a Graphic Icon 144 region and constructing a GraphicIcon 144. These operations include:

(1) Generating a low-pass filtered version, Filtered Grayscale Image 214of Grayscale Image 212 to blur and diminish any fine text and detail,preferentially leaving any bold image content relatively prominent;

(2) Generating an Edge Image 216 from Filtered Grayscale Image 214, theEdge Image 216 having high pixel values in regions where FilteredGrayscale Image 214 has certain kinds of structure, and anultra-low-resolution {fraction (1/16)}-size version of Edge Image 216,that is, Coarse Edge Image 218, for use in an initial quick search;

(3) Replacing each “pixel” of the Coarse Grayscale Image 218 by the sum(modulo register size) of a corresponding rectangle-to-there from 0,0,wherein an arithmetic combination of up to four such entries yields thesum of the corresponding pixels of the corresponding candidate rectangleof the Coarse Grayscale Image 218 and a value representing a “figure ofmerit” representing, in turn, the proportion of “image edge” in thecandidate rectangle;

(4) Searching essentially the entire Coarse Grayscale Image 218 to finda candidate rectangle of high “merit”, that is, high proportion of“image edge”, which will serve as the starting point for subsequentenlargement, reduction or modification of the region to be used inconstructing a Graphic Icon 144.

(5) Extending the left, top, right, and bottom sides of the candidaterectangle outward so long as enough edge value, as determined above,continues to accumulate so as to maintain the “figure of merit” of thegrowing candidate rectangle;

(6) Adjusting the sides of the candidate rectangle for the purpose of(a) trimming back relatively empty side areas of the image and (b)extending the sides of the candidate rectangle outward where therectangle side seems to truncate sufficiently strong structural elementsin the image; and,

(7) Adjusting the candidate rectangle to obtain a rectangle having oneof the selected aspect ratios chosen for efficient tiling of screenspace—such as 6:2, 4:3, 3:4 and 2:6, a final adjustment is made andpresented as a “fattened” version of the rectangle to iconize.

It should be noted that Find Area Program 314 has identified only thepart of the Original Image 210 that is to be re-sized, and perhapsslightly re-shaped, to form the final icon. Visual Link Capture Engine138 then generates the final Graphic Icon 144 by reference to theOriginal Image 210 in its original resolution as the source from whichthe Graphic Icon 144 is generated.

The mechanisms implemented by Find Area Program 314 also address thepossible issue that a pair of Graphic Icons 144 from different OriginalImages 210 may turn out to be sufficiently similar as to be visuallyconfusing. The mechanisms of Find Area Program 314 therefore executesteps 4 through 7 above twice to obtain an alternate region of theOriginal Image 210 to use as the Graphic Icon 144. On the second passthrough the Original Image 210, that is, through the grayscalerepresentations of the Original Image 210, the mechanisms of Find AreaProgram 314 preclude the area of the first candidate rectangle frombecoming any part of the initial candidate rectangle generated by step 4although, in the present implementation, the second candidate rectangleis not precluded from growing into and including part of, or even allof, the first candidate rectangle. The final output of the Find AreaProgram 314 mechanisms thereby include four areas, the first candidaterectangle, the first candidate rectangle adjusted to a desirable aspectratio, the second candidate rectangle, and the second candidaterectangle adjusted to a desirable aspect ratio. The choice between thetwo candidate rectangles may then be made manually by a user, forexample, in a Visual Link Capture Engine 138 as implemented in a LocalSystem 112, or automatically, for example, in a Visual Link CaptureEngine 138 as implemented in a Server System 114. In the latterinstance, the Visual Link Capture Engine 138 or, more probably, VisualLink Organizer 136, will retain a grayscale image from the newlygenerated Graphic Icon 144 selection and generation process, such asEdge Image 216 or Coarse Edge Image 218, and will compare the grayscaleimages of the two candidate rectangles with equivalent stored grayscaleimages of previously generated Graphic Icons 144.

d. Mechanisms of Find Area Program 314, Detailed Description

1. Lowpass Filter 324

Referring again to FIG. 3A for a detailed description of the structureand operation of the mechanisms implemented by Find Area Program 314, itis shown therein that Visual Link Capture Engine 138 includes a LowpassFilter 324 connected from Grayscale Image Generator 316 to constructs animage, that is, Filtered Grayscale Image 214, from each Grayscale Image212 wherein the final details of the Grayscale Image 212 image are“smeared”, that is, wherein the high spatial frequency elements aredegraded or suppressed, while the low spatial frequency elements, thatis, the larger image components, are preserved. In practice this meansgenerating each destination pixel, that is, each pixel of FilteredGrayscale Image 214, from a weighted sum of pixels of the correspondingneighborhood of the source image, that is, Grayscale Image 212. Thecorresponding program code illustrates two possible smudging filtershaving, respectively, “radii” of 1 and 2 pixels, which apply thefollowing weighting array to each target array of Grayscale Image 212pixels to generate a corresponding pixel of Filtered Grayscale Image214:

0 0 1 0 0 0 1 0 0 1 2 1 0 for radius 1: 1 1 1 for radius 2: 1 2 2 2 1 01 0 0 1 2 1 0 0 0 1 0 0

It will be understood that these low-pass filters are illustrative andrepresentative of a wide variety of sizes and patterns of such arrays ofweightings that may be employed in low pass filtering of pixel imagedata. Many alternative filters will be known and understood by those ofordinary skill in the arts and would be appropriate and, in environmentswith particular statistically frequent image characteristics, one oranother alternative filter may be preferable.

2. Edge Image Generator 326

Edge Image Generator 326, referred to as “make edge pics”, is connectedfrom Low Pass Filter 324 to receive each Filtered Grayscale Image 214and to construct two “edge” images from Filtered Grayscale Image 214,one being the corresponding Edge Image 216 and the other being thecorresponding Coarse Edge Image 218. As has been described, thegrayscale image is ¼ the linear size of the original image and the edgeimage is the same size, in pixels, as the grayscale image while thecoarse edge image is ¼ the linear size of the grayscale image, therebybeing {fraction (1/16)} the linear size of the original image.

For purposes of the present invention and the present description, theamount or degree of “edge” at a pixel in an image is the degree ofcoarse structure non-uniformity at that pixel as indicated by the rateat which the coarse structure brightness is changing in some direction.In the present implementation, Edge Image Generator 326 selects a pixelas a target pixel and examines the brightness value, as determined asdescribed previously, a selected distance to either side, or above orbelow, the target pixel, such as the pixels located two pixels above andbelow the target pixel, and determines the brightness value of the twopixels and the absolute value of the difference between the twobrightness values. Edge Image Generator 326 also determines the absolutevalue of the difference between the brightness values of the pixels theselected distance to the right and left of the target pixel. The sum ofthe absolute values of the difference in brightness values is consideredthe degree of “edge” at the target pixel. Alternatively, the maximum ofthese values may be used.

This operation may be represented graphically as illustrated in FIG. 3Band expressed as value=(|2-up−2-down|+|2-left−2-right|).

The image of edge values may be still further improved by increasing thedynamic range, transposing the image to render low values invisible tofurther computation, and clipping the value to an allowed storage range.It will be realized by those of ordinary skill in the arts, that all ofthe constants involved are adjustable to yield the best results in aparticular universe of images, the values in the present example havingbeen found to be acceptable for World Wide Web pages as typicallydesigned for 256-color colormapped display at 480-by-640 screenresolution.

In an alternate implementation, and instead of the sum of values, EdgeImage Generator 326 may be implemented to take the greatest of thesedifferences, or to use diagonally located neighbors, or neighboringpixels at greater or lesser distances away.

Other indications of being on or near an edge of coarse structure may beused instead or in addition to the above methods. For example, whether agiven target pixel is near by not exactly on a prominent edge or in themiddle of a broad line may be detected by analysis of one of the pixelpatterns represented by FIGS. 3C, 3D, 3E and 3F and their mirror imagesand 90° rotations wherein a, b, c and d are the brightness values ofpixels 3 to the left, 1 to the left, 1 to the right and 3 to the rightof the target pixel, and may be expressed for FIGS. 3C, 3D, 3E and 3F,respectively, as max(a,b)−min(c,d), max(a,b,c)−d, a−min(b,c,d) andmax(a,d)−min(b,c)

In yet other implementations, Edge Image Generator 326 may also measurefor steps up, or down, or at different distances or directions, usingthe strongest such result, or a combination of the results.

In still other implementations of Edge Image Generator 326 wherein theVisual Link Capture Engine 138 utilizes the color information availablefrom the original image, and as described previously, Edge ImageGenerator 326 may generate either a single Edge Image 216 incorporatingor combining information regarding the three color components, red,green and blue, of an Original Image 210 into a single image, or maygenerate separate Edge Images 216 and Coarse Edge Images 218 for thethree color components. It will be noted in this regard, however, thatthe use of reduced scale edge image is intended to reduce the volume ofdata that must be processed while implementations using multiple edgeimages require significantly increased image processing with consequentincreased requirements for processing speed and power or, alternatively,slower processing due to the additional data to be processed. In eitherimplementation, the color information would be used to determine edgesof image structures in three dimensional color space. Also as described,Edge Image Generator 326 may use for each entry the maximum or averagevalues of the color component values, or still other combinations. Stillother method for mapping strong image features will be useful for yetother particular applications.

Regardless of which method is used, however, to generate Edge Images 216and Coarse Edge Images 218, the essential principle is to seek a twodimensional image with high values in locations corresponding toprominent structural features in the original image, whether thestructural features are line art, such as in logos or large text, orpictures. In general, line art will have localized strong edges in thesense herein while pictures, such as photographs, tend to havecontinuing undulation of value and, while its edge maxima may be lesserthan line art, the average edge value over an area of a picture may beequivalent to that of line art. It should be obvious to one versed inthe arts of picture processing that, by adjustment of various parametersof the computation, one can strike a desirable balance between theinfluence of discrete-valued line art and that of bitmapped-continuousgrayscale pictures.

Finally, and in addition to Edge Images 216, Edge Image Generator 326also, and at the same time as it generates a ¼ size Edge Inage 216,generates a linearly 4 times coarser edge image by accumulating thevalues generated for Edge Image 216 to comprise the pixel values of thecorresponding Coarse Edge Image 218, with the accumulated values of fourEdge Image 216 pixels comprising a value for a corresponding pixel ofCoarse Edge Image 218. It should be noted, in this regard, that while anEdge Image 216 is created as a set of byte sized values, thecorresponding Coarse Edge Image 218 is accumulated as a set of 16-bitunsigned word values.

As also illustrated in FIG. 3A, Rectangular Sum Array Generator 328operates on each Coarse Edge Image 218 to generate a correspondingRectangular Sum Array 330 which, in turn, allows Visual Link CaptureEngine 138 to quickly determine, or evaluate, the “edge figure of merit”of any sub-rectangle within Coarse Edge Image 218, that is, the degreeof significant image features or structure within any sub-rectangle of aCoarse Edge Image 218 as indicated by the amount of “edge” within thesub-rectangle. It will be noted that any subrectangle must generallyhave certain minimum dimensions and must generally fall short of thefull image dimensions in each direction by at least a small minimumamount.

As illustrated in FIG. 3G, a Rectangular Sum Array 330 contains aStorage Location 332 for and corresponding to each “pixel” of thecorresponding Coarse Edge Image 218. Rectangular Sum Array Generator 328stores, in each Storage Location 332 in Rectangular Sum Array 330, thesum of the values of all pixels of Coarse Edge Image 218 containedwithin the area bounded by 0,0 as the origin at the upper left comer ofthe Rectangular Sum Array 330, which corresponds to the upper leftcorner of the corresponding Coarse Edge Array 218, and the location ofthe Coarse Edge Image 218 pixel corresponding to the storage location inthe Rectangular Sum Array 330. For example, in FIG. 3G Storage Location332 a contains the sum of the values of all pixels in the Coarse EdgeImage 218 in the rectangular area bounded on the upper left by theorigin (0,0) and on the lower right by the pixel corresponding to thelocation of Storage Location 332 a, Storage Location 332 d contains thesum of the values of all pixels in the Coarse Edge Image 218 in therectangular area bounded on the upper left by the origin (0,0) and onthe lower right by the pixel corresponding to the location of StorageLocation 332 d, and so on. It will be noted, in this regard, that thepixels corresponding to Storage Location 33 ² a, 332 b, 332 c, and so onare located immediately outside the rectangles of interest defined bythe origin an a corresponding one of Storage Locations 332.

The relative value of the total amount of “edge” structure of featuresappearing in any sub-rectangle of interest in a Coarse Edge Image 218can thereby be determined from sum values stored in the Rectangular SumArray 330 as the value sum a+sum d−sum b−sum c wherein sum a, b, c and dare the stored sum values for the comers of the sub-rectangle ofinterest. To do so, Rectangular Sum Search Mechanism 334, describedbelow, identifies the Storage Locations 332 corresponding to the pixelsin the Coarse Edge Image 218 corresponding to the upper left and lowerright comers of the sub-rectangle of interest and subtracts from the sumvalue stored therein, which is the sum of all Coarse Edge Image 218pixel values in the rectangle defined by the origin and the pixelcorresponding to that Storage Location 332, the sum values stored inStorage Locations 332 corresponding to the upper right, lower left andupper left comers of the sub-rectangle of interest. These StorageLocations 332 corresponding to the upper right, lower left and upperleft of the sub-rectangle of interest contain the sums of all CoarseEdge Image 218 pixel values in the Coarse Edge Image 218 sub-rectanglesdefined by the origin and the pixels corresponding to the upper right,lower left and upper left comers of the sub-rectangle of interest, andtherefore contained within the array sum stored in the Storage Location332 corresponding to the lower right comer of the sub-rectangle ofinterest and the result of the operation will be the sum of the valuesof the Coarse Edge Image 218 pixels contained within the sub-rectangleof interest. For example, and referring to FIG. 3G, the sum of the edgevalues of the Coarse Edge Image 218 pixels in the sub-rectangle defamedby Storage Location 332 is determined by subtracting from the sum valuestored therein the sum values stored in Storage Locations 332 a, 332 band 332 c, thereby subtracting the sum values of the threesub-rectangles defined by the origin and Storage Locations 332 a, 332 band 332 c from the sum value stored in Storage Location 332 d, therebygenerating a sum value for the sub-rectangle defined by the origin andStorage Location 332 d.

It will be noted that if the rectangle of interest extends from the leftand/or the top of the image, the calculation is correspondinglysimplified. That is, if any of the pixels corresponding to StorageLocations 332 a, 332 b, 332 c, and so on are outside of, that is, aboveor to the left of, the original total image, their rectangle-sum valuesare not looked up on the sums array, but instead the corresponding valueis understood to be zero. Alternatively, the entries of therectangle-sums array may be stored into and fetched from one position tothe right of and below where previously indicated, with the top row andleft column filled with zeros. There are then no special cases toconsider as a fetch above the top or to the left of the left edge willautomatically result in a zero value.

It is apparent that a sum value for any sub-rectangle of a Coarse EdgeImage 218 can be quickly and easily determined. It will also beapparent, as will be discussed below, that the sum values for thesub-rectangles of a given Coarse Edge Image 218 can thereafter be usedto quickly and easily determine the relative figures of merit of thosesub-rectangles wherein a figure of merit essentially indicates whether asub-rectangle contains enough edge, that is, “sum”, in relation to thesize of the sub-rectangle.

It should be noted that sum values may exceed the maximum value that maybe held in a machine word. This is generally acceptable, however,provided that overflow-traps or other such cautionary hardware orsoftware features are not activated in the system in which Visual LinkCapture Engine 138 is executing, as the sum modulo2-to-the-power-of-bits-per-word will be stored, that is, the sum startscounting over, e.g. 65534, 65535, 0, 1, 2, . . . 2. Also, in mostcomputer systems, the above arithmetic d+a−b−c will be performed modulothe same number, yielding the desired value, provided only that thehighest sum that is expected is lower than the modulus, in which casethe capture engine as implemented could add results from several suchsub-parts of the indicated rectangle so long as no part's sum exceedsthe maximum possible such sum.

Finally, the search for the optimum candidate rectangle, describedbelow, can be exhaustive, and involves tests on thousands of possiblerectangles. It will be apparent, however, that each test is relativelyfast to perform, involving calculations of only a few values each. Inaddition, in the present implementation the conversion of a Coarse EdgeImage 218 to a Rectangular Sum Array 330 is performed in a very simpleand straightforward way, computing first the top row and the leftcolumn, then filling each remaining Storage Location 332 in theremainder of the array from the values immediately above, immediately tothe left, and immediately diagonally above and to the left.

3. Rectangular Sum Search Mechanism 334

As illustrated in FIG. 3A, Rectangular Sum Search Mechanism 334,implemented as “find_start_rect” searches the Rectangular Sum Array 330corresponding to a Coarse Edge Image 216 of an Original Image 210 forcandidate regions to be potentially used in generating a Graphic Icon144 and determines a “best” first approximation to an acceptable GraphicIcon 144 region in the Original Image 210 as a Candidate Region 336A.

In this regard, in the present implementation of Rectangular Sum SearchMechanism 334 the figure of “merit” or “assurance” of each candidaterectangle is determined, as described previously, first, on the average“edge” values of the pixels, that is, on the values of the pixels ofCoarse Edge Image 218 as represented by the sum values stored in thecorresponding Storage Locations 332 of Rectangular Sum Array 330, foreach potential sub-rectangle of Rectangular Sum Array 330, and, as asecondary factor, on the size of each candidate rectangle, for thereason that a very small rectangle can easily but not meaningfullyexceed larger rectangles as regards the average “edge” value of thepixels represented therein. Stated another way, a larger region orrectangle that includes a smaller region or rectangle of higher average“edge” value is probably a meaningful enlargement of the smaller regionor rectangle.

In an alternate embodiment of Rectangular Sum Array Generator 328, thefigure of “merit” or “assurance” of a rectangle is implemented as theaverage edge value of the rectangle, as determined by the value sumsstored in Rectangular Sum Array 330, plus a constant times the perimeterof the rectangle in pixels. Clearly a variety of such alternateevaluations of “merit” or “assurance” might be used. The principleimplemented in the presently preferred embodiment of Rectangular SumArray 330, however, is that for rectangles having approximately equalvalues of “merit” or “assurance”, the larger the rectangle the better.In the reverse, for rectangles of equal size, the greater the amount ofedge (bold structure), the better.

Having identified a “best” Candidate Region 336A, Rectangular Sum SearchMechanism 334 in the presently preferred embodiment of Visual LinkCapture Engine 138 then repeats the search procedure to determine atleast a second Candidate Region 336B, thereby determining andidentifying at least two candidate rectangles from each Original Image210, as previously described. As has been and will be discussed, theidentification of two or more Candidate Regions 336 thereby allowsVisual Link Capture Engine 138 or the user to select the CandidateRegion 336 most visually representative of the original image or tocompare the Candidate Regions 336 to previously existing Graphic Icons144 and to select and retain the Candidate Region 336 most visuallydistinct from all previously existing Graphic Icons 144 without havingto repeat the entire identification process from the beginning.

In the presently preferred implementation, Rectangular Sum SearchMechanism 334 is not permitted to retrace the search through the imageby which Candidate Region 336A was identified when locating andidentifying Candidate Region 336B. Rectangular Sum Search Mechanism 334is thereby prevented from identifying the same or nearly the same regionof the image as Candidate Regions 336A and 336B as it is desirable tolocate two visually distinguished Candidate Regions 336. As such, andfor this reason, Rectangular Sum Search Mechanism 334 is designed sothat Candidate Region 336A is designated as a forbidden area of theimage and the search operation performed by Rectangular Sum SearchMechanism 334 is accordingly constrained by the designation of CandidateRegion 336A as a forbidden area to avoid overlap with a forbidden areaif in fact, a Candidate Region 336A has been previously identified.

It will be appreciated by those of ordinary skill in the arts that thereare a number of methods for constraining the Rectangular Sum SearchMechanism 334's search for Candidate Region 336B to prevent overlap, orat least significant overlap, with a Candidate Region 336A. For example,a second rectangle will not overlap a first rectangle if and only if thesecond rectangle is completely above, or completely to the left of, orcompletely to the right of, or completely below the first rectangle.Rectangular Sum Search Mechanism 334 may be implemented, however, to beconstrained from considering any rectangular region of Rectangular SumArray 330 wherein one or both of the upper left or lower right cornersof the rectangular region fall within the rectangular region ofRectangular Sum Array 330 comprising Candidate Region 336A. However,while a constraint with respect to both corners will prevent any overlapbetween Candidate Regions 336A and 336B, a constraint with respect toonly one comer would permit at least some overlap, unless furtherconstraints are imposed on the possible locations of a Candidate Region336B corner within Candidate Region 336A.

The present implementation of Rectangular Sum Search Mechanism 334, isnot constrained however, so that the fully developed Candidate Region336B will not overlap the fully developed Candidate Region 336A, as thefull extents of Candidate Regions 336A and 336B will not be known untilthe regions are fully developed, as described below. As such, an overlapmay occur between the fully developed Candidate Regions 336A and 336B,particularly when there is no acceptable but completely separate secondCandidate Region 336B.

Finally, it should be noted that search is performed by Rectangular SumSearch Mechanism 334 in what may be referred to as “{fraction (1/16)}scale space”, that is, within the dimensions and coordinates of CoarseEdge Image 218 and Rectangular Sum Array 330. s such, Rectangular SumSearch Mechanism 334, having identified Candidate Regions 336A and 336Bin terms of the coordinates and dimensions of Coarse Edge Image 218 andRectangular Sum Array 330, transforms the coordinates defining thebounds of Candidate Regions 336A and 336B, that is, the coordinates ofthe comers of Candidate Regions 336A and 336B into correspondingcoordinates in the “¼ scale space” of Grayscale Image 212, FilteredGrayscale Image 214 and Edge Image 216 by, in effect, magnifying thecoordinates of Candidate Regions 336A and 336B by a factor of four. Thefinal results of the search are thereby reported in “¼ scale space” asInitial Candidate Regions 336AI and 336B for final selection andadjustment of the selected region or regions as Graphic Icons 144, asdescribed below.

4. Grow Rectangle Mechanism 338

Grow Rectangle Mechanism 338, as implemented by “grow_rect”, cyclicallyand iteratively attempts to “grow” each of Initial Candidate Regions336AI and 336BI by extending each side of each of Initial CandidateRegions 336AI and 336BI outward. A “figure of merit” or “assurance” foreach of Initial Candidate Regions 336AI and 336BI is evaluated aftereach such extension, with the purpose of searching for larger CandidateRegion 336A and 336B rectangles with higher figures of “merit” or“assurance”.

In the initial steps of the “growing” process, a few possible extensionsare considered to the right, then upward, then to the left, and thendownward. Then, and dependent upon the results of the initial attempts,progressively larger ranges of extension are attempted and evaluated. Asa result of this process, the extents of either or both of InitialCandidate Regions 336AI or 336BI may be extended to “bridge over”possible “blank” areas, that is, adjacent areas having little or nosignificant visual structures or features, if and when there are nearbyareas having significant and presumably related visual structures orfeatures that may be preferably incorporated into either or both ofCandidate Regions 336A or 336B.

As indicated in FIG. 3A, Grow Rectangle Mechanism 338 may perform this“growing” operation using Edge Image 214 or Coarse Edge Image 216 orRectangular Sum Array 330. As has been described previously, both CoarseEdge Image 218 and Edge Image 216 contain numeric values for each pixeltherein that represent the “edge” characteristics or properties of thepixels while Rectangular Sum Array 330 contains an accumulated valuerepresenting the cumulative totals of edge information for therectangles defined by each pixel in Coarse Edge Image 218. In addition,the coordinates and dimensions of Initial Candidate Regions 336AI and336BI have been determined by Search Rectangle Mechanism 330 in terms ofthe “¼ scale” coordinates of Edge Image 216 and in terms of the“{fraction (1/16)} scale” coordinates of Coarse Edge Image 218 andRectangular Sum Array 330.

The operation of Grow Rectangle Mechanism 338 will therefore beessentially the same whether Grow Rectangle Mechanism 338 operates in “¼scale” space or in “{fraction (1/16)} scale” space, although the presentimplementation of Grow Rectangle Mechanism 338 operates in “¼ scale”space. In either implementation, Grow Rectangle Mechanism 338 willdetermine an initial figure of “merit” or “assurance” for each ofInitial Candidate Regions 336AI and 336BI wherein the figure of “merit”or “assurance” represents the amount of edge information initiallywithin the rectangles defined by Initial Candidate Regions 336AI and336BI. This information is already available for “{fraction (1/16)}scale” space from the operations of Search Rectangle Mechanism 330, asdescribed above, and an equivalent figure of “merit” or “assurance” canbe determined from the information stored in Edge Image 214 for “¼scale” space.

Grow Rectangle Mechanism 338 will then extend each of Initial CandidateRegions 336AI and 336BI by a predetermined amount in an initialpredetermined direction, such as one pixel to the right, and willevaluate the figure of “merit” or “assurance” for the resulting extendedrectangles. This process will be repeated by successive amounts, and indifferent directions, for amounts of increase and directions for whichthe figure of “merit” or “assurance” of an extended rectangle remainsapproximately the same as the initial value or increases. If the figureof “merit” or “assurance” declines for the smaller initial extensions,the process will be repeated for greater extensions, such as two, fouror eight pixels in each direction.

The final result of the operations of Grow Rectangle Mechanism 338 arethen provided as Grown Candidate Regions 336AG and 336BG.

5. Edge Adjuster 340

Edge Adjuster 340, implemented in Appendix A as “adjust_edges”, operatesupon Grown Candidate Regions 336AG and 336BG to perform a subsequentadjustment of the sides of Grown Candidate Regions 336AG and 336BG. Ashas been describe, Edge Adjuster 340 “sims away” pixels along the sidesof the rectangles having relatively little visual structure or featuresand extends the sides of the rectangles where visual or structuralfeatures extend across the sides of the rectangles, therebyincorporating the continuing visual structures or features within therectangles, and generating Edge Adjusted Candidate Regions 336AE and336BE.

Edge Adjuster 340 operates according to the criteria describedpreviously to cyclically and iteratively adjust the boundaries of GrownCandidate Regions 336AG and 336BG by one-pixel adjustments, eitherinwards or outwardson each edge, to incorporate the entirety of suchfeatures as pictures and logos within the rectangles, even when suchextensions do not meet the edge information figure of “merit” or“assurance” criteria used by Search Rectangle Mechanism 330 or GrowRectangle Mechanism 338.

As the criteria used by Edge Adjuster 340 has been discussed previously,Edge Adjuster 340 will not be discussed in further detail herein. Itwill be apparent to those or ordinary skill in the relevant arts,however, that many methods may be used to adjust the regions enclosed ineither or both of Grown Candidate Regions 336AG and 336BG to includevisual features and structures that extend outside the rectangles, or toeliminate peripheral areas of the rectangles having little visualstructure or features. The essential aspect of the operation of EdgeAdjuster 340 is to find at least one nucleus area containing strongimage material and to adjust its sides, or boundaries, so as todifferentiate visually objects contained therein from extraneous areasthat ought not to be included. For this purpose, Edge Adjuster 340operates to grow or enlarge the initial candidate area so as to includean agreeable relationship between coarse structure and size and toadjust the sides of the initial candidate area in or out to appropriatecut-off points with the goal of totally including internal visualobjects while excluding nearby parts of external visual objects.

It should be noted that the initial rectangles of Candidate Regions 336Aand 336B are determined in “{fraction (1/16)} scale” space, as describedabove, and that all subsequent operations, such as growing CandidateRegions 336A and 336B and adjusting the sides of Grown Candidate Regions336AG and 336BG are performed in “¼ scale” space in the presentimplementation of Visual Link Capture Engine 138. As such, thegranularity of the grown and adjusted Candidate Regions 336A and 336Bwill be limited to the granularity of “¼ scale” space. In alternateembodiments, however, and with additional computing time and morestorage allotted to image arrays, Visual Link Capture Engine 138 canmaintain and use half-size and/or full size arrays, alternatively orsubsequently, and perform these operations in “½ scale” or “full scale”space, yielding a 2 pixel or single pixel granularity for the finalgrowing and adjustment of Candidate Regions 336A and 336B. While thisincrease in resolution, or refinement of granularity, may result inultimately finer adjustments, the benefit may be marginal given that thelocation of coarse structure is the essential goal.

6. Aspect Ratio Adjuster and Candidate Comparator 342

Finally, Aspect Ratio Adjuster and Candidate Comparator 342,“fix_aspect_ratio” accepts proposed source image areas Edge AdjustedCandidate Regions 336AE and 336BE and enlarges Edge Adjusted CandidateRegions 336AE and 336BE horizontally or vertically to conform to one ofa set of desirable aspect ratios. As described previously, in thepresent example those aspect ratios are selected as 6:2, 4:3, 3:4 and2:6, including a border area.

Aspect Ratio Adjuster and Candidate Comparator 342 also compares theratio adjusted Edge Adjust Candidate Regions 336AE and 336BE, in themanner previously described, to select between the candidate regions,and compares the selected ratio adjusted candidate region withpreviously generated Graphic Icons 144, if any, to determine whether theselected candidate region is too similar to a previously existingGraphic Icon 144, reverting to the unselected candidate region if toosimilar a match is found with a previously existing Graphic Icon 144, oreven reinitiating the generation process if necessary. As has beendescribed elsewhere herein, and for other processes, the selectionprocess may be performed automatically by the capture engine, forexample, by performing an overall comparison between two Graphic Icons144, or manually by displaying the Graphic Icons 144 to a user forselection at the user's decision.

As has also been described, the adjustment is performed by means ofselected intermediate ratios used as aspect ratio range dividers. Forexample, if a Candidate Region 336 heighth-to-width ratio is less than4:3 and greater than 3:4, and taller than wide, the Candidate Region 336is “stretched” vertically to an aspect ratio of 4:3. Otherwise, if theratio lies within this range the Candidate Region 336 it is stretchedhorizontally to an aspect ratio of 3:4, as the applicable aspect ratiorange divider in this example occurs at 1:1. Other range dividers of thepresent example have been chosen to occur at 9:5 and 5:9.

Many aspect ratios and aspect ratio range dividers are possible,depending upon the number and range of acceptable aspect ratios forGraphic Icons 144. For example, another set of icon aspect ratios mightbe 6:2, 5:3, 4:4, 3:5 and 2:6, which would provide more advantageous“fits” for certain ranges of icons, but would result in significantlymore icon shapes and greater consequent difficulty in organizing theicons tightly in a display. Still other sets of acceptable sizes andshapes are possible, including those based on non-rectangular divisionof the ultimate display space.

As has been described, and will be understood by those of ordinary skillin the relevant arts, many methods and combinations of methods may beused to “stretch” or otherwise adjust the aspect ratios of CandidateRegions 336 to form Graphic Icons 144 or to “fit” the Candidate Regions336 into the selected aspect ratios when a given Candidate Region 336does not fit into one of the selected aspect ratios. For example, anyextra space around a Candidate Region 336 after it has been fitted intoa selected aspect ratio may be left blank in the display, which mayresult in a disorderly appearing display which, however, may beacceptable. In another implementation, the Candidate Region 336 may be“stretched” in the needed direction, horizontally or vertically, asstretches of up to 5% will scarcely be noticed and stretches of up to10% or even more will probably not be objected to by a user. In yetother implementations, the top or bottom or sides or any combinationthereof may be “trimmed” or extended as necessary, perhaps using one ofthe methods described herein above. In another method, the CandidateRegion 336 may be extended until the region reaches a recognizablebackground color, and thereafter filled with that background color,rather than including accidental and inappropriate parts of neighboringobjects.

The operation of Aspect Ratio Adjuster and Candidate Comparator 342 hasbeen described further herein above, and an implementation of AspectRatio Adjuster and Candidate Comparator 342 as “fix_aspect_ratio” andwill thereby be well understood by those of ordinary skill in the arts.As such, Aspect Ratio Adjuster and Candidate Comparator 342 will not bediscussed further herein or, and for the same reasons, will be theoperation of various implementations of Visual Link Capture Engine 138in selecting between the two Graphic Icons 144 resulting from theoperation of Aspect Ratio Adjuster and Candidate Comparator 342 on EdgeAdjusted Candidate Regions 336AE and 336BE. In this respect, however, itwill be noted that the object in selecting between the two Graphic Icons144 is to determine whether a first one of the two Graphic Icons 144 issufficiently similar to a Graphic Icon 144 already in the currentlibrary or set of Graphic Icons 144 to be visually confusing to a userand, if so, whether the second candidate Graphic Icon 144 may be moresuitable as being more visually distinctive to a user with respect tothe already existing Graphic Icon 144.

7. Reiterative Method For Finding A “Best Rectangle”

It must be noted that the “best rectangle” of an original image for usein creating a Graphic Icon 144 may also be found be means of areiterative method using essentially the same mechanism and datastructures as described above, but in a reiterative manner using aplurality of progressively coarser edge images. Although requiring theconstruction of additional edge images in the same manner as describedabove, this method involves simpler steps and is thereby faster in atleast some implementations.

According to the alternative, recursive method, the capture enginedevelops a quarter-size grayscale image as described above, for example,192 pixels wide by 144 pixels high as these dimensions are multiples of2, and from that grayscale image develops a low-pass filtered image anda first edge image. The capture engine then generates a sequence ofprogressively coarser edge images, each being reduced in resolution by 2with respect to the previous edge image of the sequence. This, forexample, will result in a 192×144 edge image, referred to as the firstedge image, a 96×72 second edge image, a 48×36 third edge image, a 24×18fourth edge image, and a 12×9 fifth edge image.

The capture engine will then perform an exhaustive search as describedabove to find the best subarea of the fifth, and smallest, edge imagethat contains none of the fifth edge image edge pixels and is at least 3pixels wide by 2 pixels high. The coordinates of the best rectangle ofthe fifth edge image are then reflected back to the fourth edge imageand 81 varients of that “best rectange” as reflected back to the fourthedge image are tested to find the “best” rectangle varient of the fourthedge image corresponding to the “best” rectangle of the fifth edgeimage. In this regard, the quantity 81 represents 3 varients on eachpossible edge adjustment of the “best” rectangle from the fifth edgeimage, that is, with each edge being pushed out one pixel, being leftalone, or being pulled in one pixel.

The coordinates of the “best” rectange of the fourth edge image that isa varient of the “best” rectangle of the fifth edge image are thenreflected into the third edge image to find the corresponding rectangletherein. 81 varients of the “best” fourth edge image rectangle that is avarient of the “best” fifth edge image rectangle are again examined inthe third edge image, again by one pixel adjustments of the edge of therectangle in the third edge image, until the “best” third edge imagerectangle is found.

This processes is repeated for successive edge images up through thesequence, moving to progressively higher resolution edge images, until afinal “best” rectangle is found.

It will be appreciated by those of ordinary skill in the relevant artsthat this alternative method thereby achieves a definition of asubrectange for use in creating a Graphic Icon 144 by successiveapproximation. It will also be appreciated that this method can, inprinciple, define any possible subrectangle of the largest edge image,except those containing the very edge row of pixels, with tests on onlya few hundred mostly smaller rectangles.

4. Detailed Description of Visual Links Organizer 166 (FIG. 4A-4I)

As has been discussed briefly above, a common problem in designingcomputer/user interfaces is to arrange the non-overlapping placement ororganization of objects, such as conventional icons, Graphics Icons 144or photographs of electronic mail recipients, of various sizes and/orshapes in a display space, such as a display window, in an efficientmanner. A further aspect of this problem in instances wherein there area large number of such displays of objects is to provide a selection ofdisplay arrangements or organizations for each given number of objectsof given sizes and shapes to allow a variety of mutually distinctivedisplay arrangements for each given number of objects of given sizes andshapes.

According to the present invention, Visual Links Organizer 166 providesa solution to these problems by providing a mechanism for efficientpacking any arbitrary collection of Graphic Icons 144 having variousdimensions and aspect ratios objects, or other visual display objectssuch as conventional icons or photographs of electronic mail recipientsof various sizes and/or shapes into a display area

It has been noted in the previous discussions of Visual Links Organizer166 that the “tiling” of many icons of various sizes and shapes, as wasillustrated in FIG. 2B, is a “bin-packing” problem of tiling geometryand that, in certain implementations, such as those with a relativelysmall number of icons per display, for example, 10 to 20, Visual LinksOrganizer 166 may be implemented by computing and tabulating at leastone arrangement for each maximally achievable icon populationdistribution. Thus, the possible arrangements for a display or asub-section of a display can be determined in advance, as pre-arrangedpacking patterns for packing, for example, two 6:2s, four 4:3s, three3:4s and one 2:6, and stored for use in a display or a sub-section of adisplay as necessary.

Therefore, and as will be described in the following Visual LinksOrganizer 166 performs this function by pre-computing and conciselystoring tabulated guides or tables for organizing visual objects, suchas conventional icons, Graphics Icons 144 or photographs of electronicmail recipients, of various sizes and/or shapes in a display space so asto speed and simplify the solution to specific object layout problems asthey arise. Despite the conciseness of these tables, however, VisualLinks Organizer 166 provides a wide variety of answers to each givenproblem, that is, a variety of visual object organization layouts foreach given number of objects of given sizes and shapes.

a. General Description of Visual Links Organizer 166

As has been discussed with reference to FIG. 1A, Local System 112includes a Visual Links Control 164 and a Visual Links Organizer 166,both of which are associated with and operate in association with WebAccess Program 162, and associated with Visual Links Control 164 andVisual Links Organizer 166 is a Visual Links Library 168 for storingVisual Links 142, which include Graphic Icons 144. As discussed, VisualLinks Control 164 operates to interface and integrate Visual Links 142,Visual Links Organizer 166 and Visual Link Capture Engine 138, ifpresent, into Web Access Program 162, and allows a user to access thetarget of a Visual Link 142 and to save a Visual Link 142 in VisualLinks Library 168 or a file. Visual Links Organizer 166, in turn, allowsa user to create and edit Visual Links 142, to store Visual Links 142 inVisual Links Library 168, and to otherwise manage Visual Links 142 andVisual Links Libraries 168. Visual Links Organizer 166, which will bedescribed in detail further below, also allows a user to organize thedisplay of Visual Links 142 in a Visual Links Library 168, for exampleby displaying Graphic Icons 144 in one or more display windows or areas.

Referring to FIGS. 1A and 4A, it is illustrated therein that a theVisual Links Organizer 166 of a Local System 112 or, for example, aServer System 114, may include a Layout Generator 410, a Layout Table412 and a Display Generator 414, for respectively generating and storingdisplay layouts and generating displays of visual objects, dependingupon the particular implementation and function of a given specificembodiment of Visual Links Organizer 166. In many Local Systems 112, forexample, as illustrated in FIG. 1A, the Visual Links Organizer 166 mayinclude only a Layout Table 412 and a Display Generator 414 and thelayouts may be provided from another system, such as another LocalSystem 112 or a Server System 114 to be stored in the Layout Table 412for use in the Local System 112.

In this regard, Layout Generator 410 is responsive to Layout DefinitionInputs 416, each including a Display Space Definition 418 a containing aspecification of a selected display area, such as 12 units by 12 units,and a Visual Object Definition 418 b containing a specification of thenumbers, sizes and aspect ratios of visual objects to be displayedtherein, for generating one or more corresponding Layouts 420 which areencoded as Plans 424, each containing a description of a layout, ororganization, of the specified visual objects in the specified displayspace. Plans 424 are then stored in Layout Table 412, where they may beaccessed by Display Generator 414 to generate a Visual Display 426 of aspecified display area containing a set of identified specific GraphicIcons 144 or other visual objects conforming with the correspondingVisual Object Definition 418 b as to number, size and aspect ratio.

Display Generator 414 is, in turn, responsive to Display DefinitionInputs 428A containing a Display Space Definition 418 a provided, forexample, by a user or by another program operating through Visual LinksControl 164 or from a Visual Link Display Memory 428B associated withVisual Links Organizer 166 for storing information relating to thedisplay capabilities of Local System 112, and specifying the displayarea to be used for displaying a set of Graphic Icons 144 or othervisual objects. Display Definition Inputs 428A also include a set ofVisual Object Identifiers 430, such as pointers to specific GraphicIcons 144 residing in Visual Links Library 168 and identifying membersof a set visual objects conforming with the corresponding Visual ObjectDefinition 418 a that are to be displayed in the defamed display area.In this regard, Visual Object Identifier 430 may be provided by a userof another program or, more probably from Visual Links Display Memory428B because a user will generally use Visual Links Organizer 166 toorganize Visual Links 142 into groups of related Visual Links 142 andwill store the identifications of the Visual Links 144 as groups inVisual Links Display Memory 428B. Display Generator 414 is thenresponsive to Display Definition Inputs 428 for indexing Layout Table412 with Display Definition Inputs 428, reading one or morecorresponding Plans 422 therefrom, and generating a Display 428 of theidentified Graphic Icons 144 or other visual objects in the specifieddisplay area and according to the selected Plan 424.

It will be appreciated by those of ordinary skill in the relevant artsthat the component parts of a given implementation of Visual LinksOrganizer 166 may depend upon the intended purpose and functions of thatimplementation of Visual Links Organizer 166. For example, and asillustrated in FIG. 1A, an implementation of Visual Links Organizer 166residing in a Local System 112 may include only Display Generator 414and one or more Layout Tables 412 as the Layout Tables 412 may beprovided from an external source, such as downloaded from a ServerSystem 114 or a Web Site 118 or otherwise provided with Visual LinksOrganizer 166 on initial installation of Visual Links Organizer 166 orlater loaded from disk. In other instances, however, an implementationof Visual Links Organizer 166 residing on a Local System 112 may includeLayout Generator 410 to allow the user of the Local System 112 togenerate Plans 422 at will. In a like manner, an implementation ofVisual Links Organizer 166 residing on a Server System 114 may includeDisplay Generator 141, one or more Layout Tables 412 and LayoutGenerator 410, thereby allowing the operators of the Server System 114to generate Displays 428 of Graphic Icons 144 residing in Visual LinksLibrary 168 for use by the operators of Server System 114 in managingthe creation an storage of Visual Links 142 therein, as well asgenerating Layout Tables 412 for local use in Server System 114 andpossibly for downloading to Local Systems 112. In yet otherimplementations, a system for generating Layout Tables 412 may includeonly Layout Generator 410 and the Layout Tables 412 generated by LayoutGenerator 410, with the Layout Tables 412 being provided elsewhere, suchas to a Local System 112 or a Server System 114 for use therein.

It will also be understood by those of ordinary skill in the relevantarts that the applications of the various implementations andembodiments of Visual Links Organizer 166 are not limited only to WorldWide Web systems and the organization and display of Graphics Icons 144but may be used in any application requiring the non-overlappingplacement or organization of objects, such as conventional icons orphotographs of electronic mail recipients, of various sizes and/orshapes in a display space, such as a display window, in an efficientmanner. The implementations of Visual Links Organizer 166 may thereforebe used, for example, to organize and display graphic representations oricons representing application programs, as in Microsoft Windows ProgramManager, or to organize and display graphic representations or icons offiles, as in Microsoft Windows File Manager, or to organize and displayicons or graphic representations, such as photographs, of recipients orlocations in an electronic mail system.

b. Definitions and Terms

The operation of Layout Generator 410 may be illustrated with referenceto FIG. 4B, which illustrates an exemplary Layout 422 generated byLayout Generator 410. In this example, the Display Space Definition 418a specifies a display area of 12 by 12 units and Visual ObjectDefinition 418 b has specified that the visual objects to be displayedtherein shall include three 6-high by 2-wide Graphic Icons 144, five4-high by 3-wide Graphic Icons 144, two 3-high by 4-wide Graphic Icons144, and one 2-high by 6-wide Graphic Icon 144, with 12 square units ofunused space occurring in the resulting Layout 422.

Before proceeding with the detailed description of Visual LinksOrganizer 116, certain of the terms and expressions used in thefollowing descriptions will be defined, and illustrated by reference toFIG. 4B. These terms and expressions include:

(a) Screen: a rectangular space allotted for tiling with icons.

(b) Coordinates (of an icon): the coordinates of a Graphic Icon 144 orother visual object in a screen is described by the coordinates statinghorizontal and vertical positions, in that order, of the top left cornerof the visual object, using the horizontal and vertical coordinates 0,1, . . . ,9,A,B, . . . Z having an origin at the top left comer of thescreen, as illustrated in FIG. 4B.

(c) Units: the basic dimensional granularity of visual object, such asGraphic Icons 144, and screen layouts.

(d) Icons: generic term for Graphic Icons 144 or other visual objectsforming the tiles for filling space, illustrated in FIG. 4B for GraphicIcons 144 having aspect ratios of 6×2, 4×3, 3×4, and 2×6 units;

(e) Layout: a specific geometric configuration of icons in a screen,referred to herein as a Layout 422.

(f) Orientations, rotations and reflections: an angular position,orientation rotation in a screen with respect to the horizontal andvertical axes of the screen and labeled with respect to the 1:00 o'clockposition, using a clock face as a coordinate system according to theconvention

so that orientation “8” specifies a reflection through a 45 degreebackslash angle.

(g) Icon set: a specified group of icons having defined aspect ratios,for example, the set illustrated in FIG. 4B is {6×2, 4×3, 3×4, 2×6} andan alternative set might be {6×2, 5×3, 4×4, 3×5, 2×6}.

(h) Fault: a cutline across a layout which divides it into tworectangles without intersecting any icons.

(i) Subrectangle: a proper subarea of a layout containing completelysome subset of its icons and no parts of other icons.

(j) Subsquare: a square subrectangle.

(k) Types: the sizes and shapes of the icon set as identified by theheights of the icons in the set, so that the types of the iconsillustrated in FIG. 4B are 6, 4, 3, 2.

(l) Population (of a layout): the number of icons of varioussizes/shapes as specified by the number of each type stated in anunderstood order, such as decreasing height, so that the population ofthe layout illustrated in FIG. 4B is specified as 3521, representing 36×2 icons, 5 4×3 icons, 2 3×4 icons, and 1 2×6 icon, wherein it usunderstood that a population may be hypothetical and may, in fact, beunrealizable in a certain size of screen, so that the population 6000cannot be realized by a layout for an 11×11 screen even though therewould seem to be sufficient space for the icons in the screen.

(m) Subpopulation (of a population): a set of icons for which the countfor each type therein is not greater than the count for correspondingtype in the population of which the subpopulation is a member, and whichis lesser for at least one type, so that the set 2421 is a subpopulationof the population 3521.

(n) Maximal population: for a specific screen size and shape, alayout-realizable pupulation such that the increase by one of any of itsicons types yields an unrealizable population for that screen size andshape.

(o) Plan: a notation, referred to herein as a Plan 424, for a layout forunderstood screen size/shape, such as the 12×12 screen illustrated inFIG. 4B and wherein the population and sequence of top left coordinatelocations of the icons are ordered by type, so that the plan for theexample of FIG. 4B is written as

3521 00 20 06, 90 94 36 66 98, 40 43, 2A,

with the understanding the commas and spaces are not required, whereinthe first number, which contains a digit for each type of icon, such as3521, defines the population of the screen by number of each type oficon, such as 3 6×2 icons, 5 4×3 icons, 2 3×4 icons, and 1 2×6 icon inthe present example, and each set of two digit numbers defines, for eachtype of icon, the coordinates of the upper left comers of the icons ofthe corresponding type, so that 00 20 06 specifies the coordinates ofthe upper left comers of the 6×2 icons, and so on. Further examples of atabulated set of layouts is illustrated in Appendix B.

(p) Inversion (of a screen's aspect ratio): a reinterpretation of aplan, such as 6×7, to its inverted aspect ratio, such as 7×6, by45-degree-backslash flip and performed by exchanging the interpretationof its x and y dimensions and interchanging all designations of aspectratios stated for the original screen, such as 6×2<-->2×6 and3×4<-->4×3, so that the two plans are inversions of each other, asillustrated in FIGS. 4C and 4D.

According to the present invention, it is necessary to generate and saveonly plans for wider-than-high and square screens for icon sets thatcontain all icons in both orientations as the correspondinghigher-than-wide, or wider-than-high, screens and the inversions ofsquare screens may be generated by inversion of the generated screens.

(q) Sublayout: a subset of a layout wherein the population of the subsetis a subpopulation of the layout so that the layout may contain thepopulation of the sublayout by occupying a part of the icon locations inthe layout with the icons of the population of the sublayout. Forexample, the population 3421 is a subpopulation of the population 3521and can be arranged in the 3521 plan by using any four of the fivelocations designated for 4×3 icons in the 3521 plan. This principle isrelated to but different from the principle that a set of one or moredifferent plans may often be generated from an original plan havingspace not occupied by icons by vertical or horizontal lateral movement,that is, sliding, of one or more of the icons of the original plan.Therefore, by visual examination of FIG. 4A it can be seen that at leastfour additional 3521 layouts may be generated from the 11×12 layoutillustrated therein by horizontal movement, or sliding, or an icon andthat more may be generated by horizontal movement of combinations of twoor more of the icons therein.

d. The Generation of Plans by A Layout Generator 410

Referring to FIG. 4E-1, therein is illustrated the process executed byLayout Generator 410 in generating at least one Layout 422 and acorresponding Plan 424 for each specified screen, such as 6×6, 6×7, 6×8,. . . 7×7, 7×8, . . . 10×6, . . . 11×12, . . . and so on. The followingwill assume that, in the presently preferred embodiment of LayoutGenerator 410, only one Plan 424 is stored for each population ofGraphic Icons 144 or other visual objects to be displayed therein,although it will be understood that in other implementations a pluralityof Plans 422 may be generated and stored for each combination of ascreen size and aspect ratio and population of Graphic Icons 144 orother visual objects to be displayed therein. It will be understood bythose of ordinary skill in the relevant arts that the following processand mechanism may be applied to the arrangement or layout of any form ofvisually displayable objects, such as conventional icons, graphic imagesof all forms and images such as photographs, in any form of display andis thereby not limited in use only to the layout of Graphic Icons 144.As such, it will be understood that in the following the term “graphicicon” is intended to refer to any form of visually displayable objectoccupying a rectangular space.

As illustrated in FIG. 4E, therefore, the process executed by LayoutGenerator 410 is generating Layouts 420 and Plans 422 is performed by aLayout Generator 410 mechanism including a Possible Layout Generator429. Possible Layout Generator 429 generates, for each screen size andaspect ratio and for each maximal population of Graphic Icons 144 orother visual objects, as specified by the numbers and types of visualobjects, as, for example, by a Display Space Definition 416 and a VisualObject Definition 420, generate all possible Layouts 420 for acombination of screen and population by trying all possible locations ofthe various types of icons of the specified population in the specifiedscreen, disallowing icon overlaps, and consider as candidates theendpoint Layouts 420 that can accommodate no further icons, that is, arefilled by the specific population.

As indicated in FIG. 4E-1, Possible Layout Generator 429 includes aSearch Forward Mechanism 431 a which searches forward on each placementof each Graphic Icon 144 or other visual object of the currentpopulation in the current screen until possibilities have beenconsidered, wherein a search forward is an attempt to place a next iconfrom a list of icons of the current population into the current screen.

Possible Layout Generator 429 further includes a Backtrack Mechanism 431b connected from and interoperating with Search Forward Mechanism 431 awhich backtracks on each placement of each Graphic Icon 144 or othervisual object of the current population in the current screen until allpossibilities have been considered, wherein a backtrack is the removalof the most recently placed icon and an attempt to place a next iconfrom the list of icons of the current population into the screen and, ifno subsequent icon can be placed after removal of the most recent icon,a removal of the next previously placed icon and an attempt to place anext icon from the list of icons of the current population.

A Layouts Storage Manager 432 then stores each Layout 422, that is, Plan424, in a Layout Generation Memory 434, shown in FIG. 4A, for subsequenttesting and selection. In this regard, a Layout Test Mechanism 436 isconnected from Layout Generation Memory 434 through Layouts StorageManager 432 to read each Layout 422 from Layout Generation Memory 434and to test each new resulting Layout 422 against all other Layouts 420previously generated in Step 430 and stored in Layout Generation Memory434.

Layout Previous/New Selector 438 is connected from Layout Test Mechanism436 and selects between a new Layout 422 and a previously existingLayout 422 according to a predetermined criteria. For example, and inthe present embodiment, if a new Layout 422 has a subpopulation of aprevious Layout 422, Layout Previous/New Selector 438 discards the newLayout 422. If a previous Layout 422 has a subpopulation of a new Layout422, Layout Previous/New Selector 438 replaces the previous Layout 422,unless disqualified in a following step, as described below. If aprevious Layout 422 has a subpopulation of the new Layout 422,Layout/Previous Selector 438 discards the new Layout 422 and, if apreviously existing Layout 422 has the same population as a new Layout422, Layout Test Mechanism 436 selects between the previous and newLayouts 420 by discarding one or the other by either (1) randomselection between the previous and new Layouts 420 having the samepopulation or (2) selecting the Layout 422 having the greater asymmetryas potentially generating the larger number of different Layouts 420through a subsequent step of usage-phase modification, described below.

At this stage in the process there may still survive Layouts 420 havingpopulations that are subpopulations of other surviving Layouts 420, suchas a Layout 422 that eliminated a previously generated Layout 422, thatis, a Layout 422 with a subpopulation of its own, that may in turn mightalso disqualify others Layouts 420 previously generated which also havesubpopulations of that Layout 422's population. A final selection ofLayouts 420 may therefore be performed by a Layouts Comparator 440wherein the populations of each pair of surviving Layouts 420 arecompared and Layouts 420 having populations that are subpopulations ofLayouts 420 are discarded. Alternately, this process may be performed bycomparison of each pair of Layouts 420 in Layout Test Mechanism 436,depending upon which implementation proves most efficient in a givenembodiment of Layout Generator 410.

Finally, the surviving Plans 422 encoding the final surviving Layouts420 are tabulated and stored in are tabulated, as demonstrated in one ormore Layout Tables 412. In this regard, it should be noted that in apresently preferred embodiment of the invention Plans 422 are grouped inLayout Tables 412 by screen size and aspect ratio, often with eachLayout Table 412 continuing the Plans 422 for a given screen size andaspect ratio, and that Layout Tables 412 may then be indexed, first, byavailable display, or screen, size and aspect ratio, with the selectedLayout Table 412 then being indexed by the icon population to readtherefrom a Plan 424, if available therein. In other implementations,the Plans 422 may be grouped first by populations and then by screensize and aspect ratio.

It should be noted that in as much as it is often necessary to generateno more than one, two or three Layouts 420 for each maximal achievablepopulation for a particular screen, Step 430 may be implementedalternatively by first searching for the full area occupancy populationsfor the screen for each possible maximal set of icons that would fillthe screen area, with Layout Generator 410 searching for a fitting ofeach of the sets of icon shapes into the screen area. Then, for eachpopulation that cannot be realized, Layout Generator 310 searches forLayouts 420 for each possible subpopulation of that population thatcontains one less member, repeating the search with successivesubpopulations that are each reduced a different icon type and, in eachbranch of the search, repeating the search with progressively reducedsubpopulations until a subpopulation is found that can be fitted intothe screen area, to find all possible subpopulations of the originalpopulation that can be fitted into the screen area. The resultingLayouts 420 are then culled by comparison to previously generatedLayouts 420, as described above.

It will be recognized by those of skill in the relevant arts that theprocedure of Steps 430 through 440 described above may requiresignificant computing power to time to complete due to the large numberof possible screen sizes and possible plans that must be searched foreach possible population. The method of the present invention, however,and as described below, provides additional initial steps to increasethe speed and efficiency of the generation of Plans 422.

As illustrated in FIG. 4E-2, therefore, and according to the presentinvention, mechanism described above with regard to FIG. 4E-1 mayfurther include mechanism for performing the following operations inassociation with or preceding the operations performed in PossibleLayout Generator 429 to generate patterns of various instances of iconlayouts in various screens and to provide the resulting patterns for usethe process described above.

In this regard, and referring to FIG. 4E-2, Layout Generator 410 mayfurther include a Screen Map Generator 442 for generating, for eachscreen size and aspect ratio up to a maximum screen height (or width) ofX units, a corresponding Screen Map 444 at a resolution, for example, ofY bits-per-unit-square in as many X bit words as the screen is wide (orhigh) wherein X is selected as a convenient word length for processingin the system in which Layout Generator 410 is implemented and Y is, forexample, the smallest dimension, in bits, of any icon. For example, manycomputers operate efficiently with 16 bit words and, accordingly X isselected as 16 in a presently preferred embodiment of Layout Generator410.

The Screen Maps 444 are then provided to an Instance Patterns Generator446 that, for each icon type, that is, icon size and aspect ratio,generates as many screen-size “Instance” Patterns 448 as there arepossible locations in the Screen Map 444 for that type of icon, usingspecial encoding for clear space at top and bottom between the placementof a icon and a side or the top or bottom of the Screen Map 444 asnecessary. This process of determining whether a given instance of anicon of a given type into the Screen Map 444 will fit, and placing orremoving it, is thereby comprised of fast logical AND, OR andEXCLUSIVE-OR operations, respectively.

It will be noted that with regard to allowing clear space along any sideof the Screen Map 444 that if there is no icon type under considerationwith minimum dimension less than two, then it is unnecessary to considerinstances, or placements of the icon in the Screen Map 444, havingone-unit space between the icon and an edge, unless it is necessary dueto the shapes of an icon and the screen, as in the placement of a 2×6icon on a 7×7 screen. Therefore, in making instance patterns of iconpositions, Step 446 omits near-but-not-on-side locations except wherenecessary.

The result of these operations is thereby at least one Instance Pattern450, an example of which is illustrated in FIG. 4F, containing asufficiently complete set of Locations 450 for each icon type in thescreen represented by Screen Map 444, each Location 448 representing apossible instance of the icon type in the given screen. The example ofan Instance Pattern 450 illustrated in FIG. 4F shows a sufficientlycomplete set of Locations 450 for 3×4 icons in a 7×9 screen wherein eachLocation 448, each of which is indicated by a *, identifies the locationof the upper left corner of a 3×4 icon for a possible placement, orinstance, of an icon of that type in the given screen.

Finally, a Location Instances Enumerator 452 enumerate the instances ofthe icon of the given type in the screen in order of proximity to thescreen corners for each screen and icon type to be considered in acorresponding Instance Enumeration 454, and provide the InstanceEnumerations 454 for use by Possible Layout Generator 429.

As has been discussed above, the process executed by Possible LayoutGenerator 429 involves a significant number of reiterations of iconplacement steps and backtrack steps, which in turn entails theinadvertent rediscovery of members of sets of icons far down through theprocess steps that conflict with one another. As such, the processes andoperations described above may be further enhanced for the case of largescreens, such as 12×12 or larger, by a method wherein variously locatedinstances of each shape of icon to be placed in a screen are sorted intodisjoint sets according to their locations in the screen and each set istreated as a “supericon”, the supericons then being used in the screenfilling process described above.

As illustrated in FIG. 4E-3, Layout Generator 430 may thereby furtherinclude a Screen Spatial Division Generator 456 that determines, for alarge screen such as 12×12 or larger, the various possible locationinstances of each type of icon with respect to nine spatial divisions ofthe screen, which are illustrated in FIG. 4G for a 12×12 screen. As maybe seen therein, the screen is divided into four quadrants by horizontaland vertical axes of symmetry and four of the spatial divisions areformed by the four corner quadrants formed by the two axes of symmetrywhile the right and left halves of and along the horizontal axis ofsymmetry, the upper and lower halves of and along the vertical axis ofsymmetry and the intersection point of the axes of symmetry in thecenter of the screen form the remaining five spatial divisions. The ninesets correspond to Spatial Divisions 458 a, 458 b, 458 c and 458 d,which correspond to the four comer quadrants, Spatial Divisions 458 e,458 f, 458 g and 458 h, which correspond to the halves of the two axes,and Spatial Division 458 i, which corresponds to the intersection pointof the two axes.

Thereafter, a Location Instances Sorter 460 sorts, for and according toeach type of icon, the location instances of the icon in the screenaccording to their positions with regard to nine spatial divisions ofthe screen.

Then, and for each of the nine sets, a Location Instance Search andCompose Mechanism 462 searches for and composes compatible, that is,non-overlapping, combinations of icon location instances, for example,by operation of Layout Generator 410 or by presenting possible choicesto a human operator, and a following Super-object Generator 466designates each compatible combination as a Supericon 464, with thepurpose of filling the screen as completely as possible with Supericons464.

For each of the nine sets of icons, and in principle, all combinationsof 0, 1, 2, or 3 or more icons may be considered as Supericons 464. Thesearch for such combinations may be restricted, and the searchaccordingly shortened, according to certain principles.

First, for Spatial Divisions, or sets, 458 a, 458 b, 458 c and 458 d,which occupy the four comer quadrants of the screen, all combinationshaving at least one quadrant with zero members may be discarded as nomaximal population can result from a combination having an empty comerquadrant. Further, in any pattern in which a comer unit of a quadrant isnot occupied, one of the icons near the comer may be moved, ortranslated horizontally or vertically, into the empty comer quadrant,and that pattern will result at least one other point in the search.That is, the two instances, with the icon near the comer and with theicon in the comer, are essentially equivalent so that the icon may aswell be placed in the comer rather than considering the two instancesseparately.

Second, there will most probably be one or more combinations whereinthere are “slits” one unit wide between two icons which cannot be filledby vertical or horizontal translation of one of the icons, and which mayin fact impede the placement of other icons. Also, the configurationthat would result from moving the icons to close the “slits” will mostprobably occur elsewhere in the search and, as a result, as no maximalpopulation can result from a combination generated by vertical orhorizontal translation of an icon to eliminate a “slit” space betweentwo icons. Such combinations are therefore preferably discarded fromconsideration.

It will be noted that the above two situations are instances whereinthere are two equivalent combinations, one where there is a non-fillableslit and one where an icon has been moved to fill the slit and resultsin an equivalent non-fillable slit. In these instances it is notnecessary to consider both of the equivalent combinations, but onlythose combinations wherein moving an icon results in the locking out ofan icon placement possibility. For example, and referring to FIG. 4H, ifthe 4×3 icon at the upper left were one unit further right, no newcombinations or placements would result than what is possible with theicon in its present placement, so it is not necessary to consider thecombination with the icon moved one unit to the right. In the instancewherein the upper left 6×2 icon and the 4×3 icon are reversed, however,with a unit space between the icons, a movement of the 6×2 icon to theleft precludes a 3×4 icon immediately under the 4×3 from being placed,so that this instance should be considered.

Finally, some combinations occupy exactly the same areas and haveexactly the same subpopulations as other combinations and are thereforeequivalent and may be discarded from consideration.

The search for such combinations may be restricted and the culling ofcombinations according to the above principles may be implemented as bySupericon Designator 466 operating in association with Location InstanceSearch and Compose Mechanism 448, or may be performed manually bypresenting possible combinations to a human operator for decision. Itshould be noted that experimentation has shown that the application ofthese principles will result in approximately ¾ of the possiblecombinations surviving the culling process, so that the generation andsearch process can accordingly be reduced to approximately ¾ to theninth power of the original time required, that is, to approximatelyless than a tenth of the original time required to generate and searchlayouts for large screens.

Lastly with regard to the use of Supericons 464 to reduce the number ofreiterations of icon placement steps and backtrack steps, it will beapparent from the above discussions that certain sets of combinationsgenerated as described above will result in sets of combinations thatare rotational or reflectional equivalents to one another. For example,each of several starting points of a search for combinations may startwith a set of combinations with one combination from each of the cornerquadrants of the screen, that is, a 458 a, 458 b, 458 c and 458 dcombination wherein each of the combinations, when appropriately rotatedor reflected, is equivalent to the other combinations. This may beillustrated as in FIGS. 4H and 4I wherein it is apparent that the twostarting sets of combinations are equivalent because the subpopulationand covered area pattern of FIG. 4I is the same as that of FIG. 4H whensubjected to a rotation about the right half horizontal axis.

As also discussed above, it is anticipated that the final usage ofLayouts 420 to generate screen displays anticipates that at leastcertain Layouts 420 will be rotated or reflected to generate yet otherLayouts 420, so that the end result of the entire process could be thegeneration of equivalent Layouts 420.

As a consequence, the search for such combinations may further berestricted and the generation of combinations enhanced by a processwherein the starting combinations are pruned, or culled, or any set ofstarting quadruple combinations that are rotational or reflectiveduplicates of already generated starting combination quadruples.

The speed and efficiency of Possible Layout Generator 429 and thus ofthe subsequent processes described above may also be increased byfurther mechanism associated with Possible Layout Generator 429 or asubsequent process wherein a large screen is generated as a combinationof two or more smaller screens.

Therefore, and as illustrated in FIG. 4E-4, Layout Generator 410 mayfurther include a Comer Area Examiner 470 connected from and to PossibleLayout Generator 429 which, for very large screens, examines the comerareas of each Layout 422 as it is generated and retain only Layouts 420whose comers are filled in unique ways, that is, discard one or any pairof Layouts 420 if the patterns of similar icons filling their comers arerotational or reflectional equivalents, that is, if one is a 5, 7, or Breorientation of the other. Comer Area Examiner 470 may also apply thecriteria that all four comers of a Layout 422 be occupied.

Finally, the speed and efficiency of Layouts Generator 430 and thus ofthe subsequent processes described above may also be increased byadditional processes primarily intended for use with large screens, forexample, greater than 12×12. In these additional processes, Layouts 420are composed by dividing the screen into two or more smaller screens,generating solutions for the smaller screens, and generating thesolution for the original large screen by combining the solutions forthe smaller screens. In this embodiment, as illustrated in FIG. 4E-5 andprimarily for use with large screens, Layout Generator 410 will furtherinclude a Screen Sub-Area Divider 472 connected from Possible LayoutGenerator 429 which divides each screen, such as a 13×12 screen, intotwo or more smaller screens or sub-areas, for example, (n)×12 and(13−n)×12 wherein n=5, 6, 7. Screen Sub-Area Divider 472 then providesthe smaller sub-area screens to Possible Layout Generator 429, whichuses each sub-area screen independently as a display area for thegeneration of corresponding Layouts 420.

Then, a Screen Sub-Area Combiner 474 operates in conjunction withLayouts Comparator 440 to compose Layouts 420 for the original full sizescreen by joining the Layouts 420 generated for the smaller sub-areascreens, adding the populations of the smaller screens, transposing thecoordinates of the smaller screens into coordinates of the large,combined screen and test for duplication and inclusion relationshipsamong the resulting Layouts 420, as described previously.

Alternatively, the Layouts 420 may be generated for the smaller screensas described above and a Layout 422 may be generated for the originallarger screen by Location Instance Search and Compose Mechanism 462wherein Layout Generator 410 or another program searches Layout Tables412 for a combination of smaller screens that may be combined into ascreen having the dimensions and population of the large screen, and socombines the smaller screen Layouts 420 into a Layout 422 for the largescreen.

The above described process of generating a layout for a large screen asa combination of layouts for smaller screens may be illustrated with theaid of FIG. 4J wherein an M×N Screen 476 is divided two SubScreens 476Aand 476B having dimensions, respectively, of M×A and M×B and which maybe side by side, as illustrated in FIG. 4J, or one above the other.Layouts 420 for SubScreens 476A and 476B that are useful in generating aLayout 422 for Screen 476 may be generated and tabulated in advance byconsidering the generating all pairs of sublayouts for screens havingdimensions of M×A and M×B wherein A+B=N. Layout Generator 410 cangenerate a very large number of populations that can be arranged in M×Nscreens by adding various A+B populations. As described above, LayoutGenerator 410 may discard any resulting population that is asubpopulation or equivalent population to one already generated, and maydisqualify those layouts whose populations are subpopulations of a newlayout.

It will be apparent from the above discussions that the method ofconstructing Layouts 420 for screens as combinations of Layouts 420 forsmaller screens that can be combined into the larger screen can beextended to combinations of greater numbers of smaller screens, asillustrated in FIGS. 4K, 4L and 4M, which respectively illustrate aLayout 422 for a 7×7 screen as a combination of Layouts 420 for four 4×3and 3×4 screens, a Layout 422 for a 14×14 screen as a combination ofLayouts 420 for 16 4×3 and 3×4 screens, and a 14×14 screen as adifferent combination of Layouts 420 for 16 4×3 and 3×4 screens. It isalso shown in FIGS. 4K, 4L and 4M that the resulting screen need nothave “faults” or lines of rotational or reflective symmetry among thesmaller screens comprising the larger screen.

Finally in this respect, the Plans 422 for composite layouts for largerscreens comprised of two or more smaller screens may be tabulated intoLayout Tables 412 by listing the Plans 422 by its subparts, either byassigned serial numbers or by populations, rather than by the membericons, and that the definitions of the composite Plans 422 can cascade.

Also, the previously discussed methods for generating additional Layouts420 from a Plan 424 stored in a Layout Table 412 by rotation orreflection of the stored Plan 424, that is, its corresponding Layout422, may be used to generate additional composite Layouts 420 byrotation or reflection of the stored composite Plan 424.

e. Using Plans 422

As discussed above, a user of a Visual Links Organizer 166, and inparticular Display Generator 141 and one or more Layout Tables 412, willgenerate requirements such as “for a 12×10 screen, provide a plan forplacing population 3321.” Such a need is satisfied (or not) by a linearscan of the appropriate Layout Table 412 for the first Plan 424 thatcontains at least the desired population. The result of the search iseither a Plan 424 or a failure, the latter meaning that the statedpopulation of icons cannot be arranged on specified screen. although itshould be noted that if the specified screen is inverted with respect toscreen for which a Plan 424 exists, the request may be satisfied byinverting the Plan 424 for the screen for which a Plan 424 is found.

As described above, a Plan 424 is a specific geometric layout for thestated numbers of icons of the specified types and for the specifiedscreen and the user may use the Plan 424 unaltered or, for the sake ofvariety in the appearances of numerous similar layouts, may performvarious alterations to the Plan 424 to vary the visual appearance of thePlan 424.

For example, criteria for the search for a Plan 424 is satisfied if thesearch population is a subpopulation of an existing Plan 424. Therefore,and if the user does not have additional icons of the number and typesnecessary to fill the available space in the Layout 422 represented bythe Plan 424, there is at least some flexibility as to which of the iconpositions to use and which to leave blank, thereby varying the visualappearance of the Layout 422. The choice of which icons positions tofill may be made, for example, at the individual discretion of the user,for example, using “drag and drop” or “point and click” operations toassociate icons with locations, or the locations may be assigned bymeans of a random number generator, perhaps, in the instance of reuse ofthe same Layout 422, using the assignments for an earlier use of theLayout 422 as a seed for the later uses of the Layout 422.

In further example, the visual appearance of a Layout 422 may be variedby rotating or reflecting the Layout 422 about, for example, thevertical or horizontal axis or the 1, 5, 7, B orientations of the Plan424, either at random or by user selection. Also, if the specifiedscreen is square and the population is of the form a b b a, the Layout422 may be reoriented about the 2, 4, 8, and A axes, withreinterpretations of the icon types, such as 6-high<-->2-high and4-high<-->3-high.

In a like manner, if the Layout 422 contains a vertical or horizontalfault, the placement of the subrectangles defined by the fault may beexchanged or, if there exists s subrectangle in Layout 422, the area ofthe subrectangle may be reoriented by along the 5, 7, or B axes or, ifthere is a subsquare and its subpopulation is of the form a b b a, thenthe icons of the subpopulation of the subsquare may be reoriented in asimilar manner.

Finally, if there is empty space along any side of an icon, the icon maybe moved vertically or horizontally to occupy the empty space.

It will be appreciated by those or ordinary skill in the relevant artsthat these processes are easily programmed so as to happenautomatically, and require little time to execute. It will also beapparent that it is useful to perform these operations in various ordersand iteratively in various combinations, so long as new geometriclayouts result.

5. Description of Visual Links 142 (FIGS. 5A-5G)

As previously discussed, for example, with reference to FIG. 1B, aVisual Link 142 may generally include a Universal Resource Locator (URL)128, a Title 130, a Text 132, and a Date 134, and may include Keywords136, and will include a Graphic Icon 144. As has been described indetail above, a Graphic Icon 144 is generated or extracted from thegraphics and text information present in the corresponding HTML documentand which serves as a visual representation to the user of theassociated document or file. The role of Graphic Icons 144 may also befulfilled by graphic representations of various forms of files,locations or recipients on a network, such as photographs of electronicmail recipients.

In addition to representing a document, file, location or recipient, forexample, in a visual index of documents, files, locations or recipient,a Graphic Icon 144 is a computer programming construct which permits asmall bitmapped picture, that is, a Graphic Icon 144, to serve as anactive button on the screen of a World Wide Web browser, electronic mailprogram or other application program. A user may then initiate access tothe document, file, location or recipient for a corresponding purpose by“clicking” on the button. As has been described, Graphic Icons 144enable users to more readily and easily recognize and remember a Website, document, file, location or recipient, by providing arepresentative visual representation of the Web site, document, file,location or recipient, than do text indicators or, for example, ASCIIstring universal resource locators (URLs).

As illustrated in FIG. 5A, a Visual Link 142 concatenates, combines, orotherwise associates, a normally encoded bitmapped picture, that is, aGraphic Icon 144, with a set of Visual Link 142 data, referred tohereafter as a Visual Link Dataset 500, and includes a means forassociating the Graphic Icon 144 and the Visual Link Dataset 500. Thefollowing will describe each of these elements in turn.

As has been described, the active button bitmapped picture is comprisedof a Graphic Icon 144, which is, for example, on the order of 100 by 100pixels in size and is intended to be encoded in any of the commonlyknown means of bitmap encoding, which include, but are not limited to,JPEG, GIF and Microsoft BMP. The image data may be compressed by any ofthe methods thereby supported in each case and, as used for the purposesand uses of Visual Link 142, the basic structure and detailed content ofthe picture file remains unaltered.

A Visual Link Dataset 500, in turn, is an ASCII file or the equivalent,such as an encrypted alphanumeric file, which contains, mostessentially, a URL 128 which, in turn, is an ASCII character string, forexample,

http://www.netwavelink.com/pointreview.HTML. A minimum Graphic Icon 144dataset for this example might therefore appear as illustrated in FIG.5B wherein it is shown that this minimal Visual Link Dataset 500includes three Designators 502, respectively indicated as Graphic Icon144=, URL 128=

http://www.netwavelink.com/pointreview.HTML, and END 504=

Each Designator 502 is a body or item of data related to the Visual LinkDataset 500 and indicates the nature of the following ASCII data, ifany, and ends with the ‘=’ character. ‘URL 128’ of the correspondingDesignator 502 illustrated in FIG. 5B thereby represents the URL 128 ofa corresponding document, site, file, location or recipient or the like,with the “=” indicating the end of the designator.

While the URL 128 Designator 502 is the crucial item for contacting aweb site, the user will often find it helpful if additional relevantinformation is stored in the Visual Link Dataset 500, thus makinginformation available immediately when and if the Graphic Icon 144 isstored locally. A typical Visual Link Dataset 500 with such additionalinformation may appear as represented in FIG. 5C wherein the Visual LinkDataset 500 is illustrated as including the Designators 500 representedas Graphic Icon 144=, URL 128=, Text 132 including a Subject 506=and anAuthor 508=, KeyWords 136=and an END 504=. It should be noted that onlyGraphic Icon 144=, URL 128=and END 504=are essential and that only URL128=absolutely requires an argument, that is, the ASCII character stringor equivalent comprising the URL 128.

A further example of a rather extensive Visual Link Dataset 500 mightbe:

Graphic Icon 144=Version 1.0

KeyWords 136=web browser interface icon button visual internet

bitmap compression thumbnail

Subject 504=web_browsing

Category=browsers

Company=Netwave, Inc.

USPS=440 Middlesex Road

Tyngsboro MA 01879

Phone=(603) 649-8633

FAX=(603) 649-8699

E-Mail=gmiliefsky@netwave.com

Description=Provider of “Visual Links Objects” for making cyberspaceeasier.

Comment=

PrevGraphic Icon 144=

URL=http://www.netwavelink.com/pointreview.HTML

SerialNo=0000000045

SeeAlso=http://www.foobar.edu/santa.clause

SeeAlso=http://www.patriot.mil/big.guns

Author=Gary Miliefsky

Created=11/10/96

LastMod=11/14/96

END 504=

NOTE: This dataset (11/14/96), is intended only for the purpose ofdebugging the various software modules that Netwave will soon export. ITIS NOT TO BE DISTRIBUTED TO OUTSIDE PARTIES EXCEPT AS AN EXAMPLE, ANDTHEN ONLY TO THOSE WHO HAVE SIGNED A NON-DISCLOSURE AGREEMENT. Althoughthis dataset was current as of Nov. 17, 1996, there may be yet newerversions, with slightly different names. Look for all files with namessamp*.*.

It will be noted that this example includes text appended at the end,which may be helpful for developers or other users, and that the END504= Designator 502 is the end of data of significance to the GraphicIcon 144 parser and hence to all computer programs that interrogate orotherwise use Graphic Icons 144.

It should also be noted that in the present implementation, Designators502 are case-insensitive, meaning that any upper case letters and theircorresponding lower case versions are interchangeable. In addition,there may be alternative forms for at least some Designators 502. Forexample, in order to save storage space and transmission time, at leastsome Designators 502 may be represented by four-character abbreviations,such as KEY= for KeyWords=. In other examples, the following Designators502 may be represented by the corresponding following abbreviations:

Graphic Icon 144= by Icon 144=;

Comment= by COM=;

Remark= by REM=;

Date= by DAT=;

Author= by AUT=;

Contact= by CON=;

Key Words= by KEY=; and,

Subject= by SUB=.

The above illustrated group or set of Designators 502 is intended to bea growing set, with additional Designators 502 added as found necessaryor advantageous for general use. It may, for example, be desirable toadd a “rating system” designator whose arguments indicate the depictionor or reference to sex, nudity, violence or language unsuitable foryounger age audiences. There may also be designators whose argumentshold information dealing with encryption or permissions, whereby certainparts of the dataset may be read or interpretable only by userspossessing appropriately entitling decryption methods or passwords. Forexample, the site referenced by a URL may thus be visible or accessibleonly to privileged users.

It is also intended that specific users may add Designators 502 to meettheir specific needs, such as failed= for the cumulative count offailures of the site to respond, found= for the count of successes atdownloading from the site, and FSLS= for the count of failures since thelast successful access to the site.

It is intended that Visual Link Datasets 500 may be parsed, interpretedand use by relevant systems and programs from a variety of sources. Assuch, it is further intended that Visual Link Datasets 500 conform to acommonly accepted set of at least minimum rules and definitions so thatany given Visual Link Dataset 500 may be used, at least to the minimumnecessary extent to meet the intended purpose, by any system or programconforming to these rules and definitions. The following explicit rulesand definitions, or an alternative but generally similar set of rulesand definitions, are exemplary of such a set of rules and definitions.

Accordingly, the following are definitions used in a presently preferredimplementation of Visual Link Datasets 500:

A WHITESPACE is a contiguous string of non-printing non-ink characters,most notably space, tab, carriage return, line feed, and the implicitlyunderstood beginning-of-file and end-of-file.

A TOKEN is a contiguous string of visibly-printed ASCII characters thatappear between successive whitespaces.

A DESIGNATOR 502 is a token that ends with the character ‘=’.

An ARGUMENT is a token not ending with ‘=’ appearing somewhere betweensuccessive designators.

An ENTRY is logically a line of Visual Link Dataset 500 code consistingof a Designator 502 and the following argument(s), if any, up to but notincluding the next designator. According to a present implementation ofVisual Link Datasets 500, the arguments of a designator need not appearon one line, that is, the whitespace separating arguments may be orinclude carriage return and/or linefeed.

In a like manner, the rules and practices for a presently preferredembodiment of the present invention include that a Visual Link Dataset500 is comprised of:

a Graphic Icon 144= <argument(s)>;

<entry or entries other than the URL 128=>;

a URL 128= plus one argument in the form of a URL-format web address;

<entry or entries other than the URL 128=>;

END 504=; and,

<text that will be ignored by all programs,

wherein the items delineated in < > above are regarded as optional.

In addition, it is presently good practice is to avoid arguments thatend in ‘=’, and when this is impossible, then to end the argument with“=” written in the form ‘{circumflex over ( )}=’, correspondinglywriting the character ‘{circumflex over ( )}’ in the form “{circumflexover ( )}”.

In other usages, the minimal Visual Link Dataset 500, like that depictedabove, may absolutely require some designator and related argumentsother than a URL=, such as an e-mail address, or phone number or postaladdress.

According to the presently preferred embodiment, the order, number ofentries, and their content in a Visual Link Dataset 500 is arbitraryexcept that

(1) there must be one and only one Graphic Icon 144=, URL 128=and END504=,

(2) the very first token of a Visual Link Dataset 500 must be GraphicIcon 144=, except that this token is not required in the Visual LinkDataset 500 version wherein the Graphic Icon 144 and the Visual LinkDataset 500 are contained in a single file as the dataset need not pointto the icon,

(3) the first and probably only argument of the URL 128= token must be,or at least thought to be, a bona fide URL 128, and

(4) the END 504= entry is the last entry or token of the file that willbe recognized and treated by support programs that parse or use VisualLinks 142.

Also, and for purposes of human creation, examination and/or editing, itis preferable that a Visual Link Dataset 500 be comprised of humanreadable characters, such as ASCII printing characters, blank, carriagereturn and line feed.

As has been described herein, a Visual Link Dataset 500 may be generatedautomatically by a capture engine, such as a Visual Link Capture Engine138, whereby Web sites are interrogated and the Graphic Icon 144 bitmappicture selected and other data gleaned by downloading and scanning theHTML file from each site. Since a Visual Link Dataset 500 is encoded asan ASCII file, however, a Visual Link Dataset 500 may alternatively begenerated manually, for example, by a person using a text editor such asa normal word-processing program.

Having considered Graphic Icons 144 and Visual Link Datasets 500 hereinabove, the following will consider the means for association of aGraphic Icon 144 and a Visual Link Dataset 500 to construct a VisualLink 142. Association of Picture and Graphic Icon 144 Dataset.

As discussed, a Visual Link 142 consists essentially of a picture, thatis, a Graphic Icon 144, and a Visual Link Dataset 500, which must beassociated with one another to form an entity referred to herein as aVisual Link 142. and a Graphic Icon 144 dataset This may be accomplishedin several ways, as illustrated in FIGS. 5D through 5G., the presentlypreferred method being illustrated in FIG. FIG. 5F.

Referring first to FIG. 5D, in this embodiment the Graphic Icon 144picture is described in a typical graphic or image file format, such asJPEG or GIF, with pixel data compressed or not, and is this file iscompletely isolated from the Visual Link Dataset 500, forming a separatefile, and is unaltered. The association between the Graphic Icon 144file and the Visual Link Dataset 500 file is maintained on a higherlevel, for example, in a database or link file structure, identified inFIG. 5D as Link 510 that contains at least the relative or absolutePathnames 512A and 512B of the two files, or the equivalent.

Advantages of this embodiment is that the picture file may be used inany and all of the possible ways that are relevant to the format of theencoded bitmap picture, as can be the Visual Link Dataset 500 file, andthe Graphic Icon 144 picture file and the Visual Link Dataset 500 fileare individually created or modified more easily. A disadvantage of thisembodiment, however, is that three items, the Graphic Icon 144 file, theVisual Link Dataset 500 file and the database or file structure relatingthe two, must be maintained in agreement, and that more care and/orutilities are thereby needed to maintain between the components of theVisual Link 142.

An alternate embodiment is illustrated in FIG. 5E wherein it is shownthat the Graphic Icon 144 picture file and the Visual Link Dataset 500file are again separate files but that the association between them isindicated in a Designator 502 in the Visual Link Dataset 500, such as‘PIC 514=’. In this embodiment, the argument of PIC 514= is the absoluteor relative pathname, or equivalent for the Graphic Icon 144 picturefile, thereby eliminating the database of file structure used in theembodiment of FIG. 5D to relate the Graphic Icon 144 and the Visual LinkDataset 500. The advantages of the embodiment illustrated in FIG. 5E arethe same as for the embodiment illustrated in FIG. 5D, and thedisadvantages are similar in that there are still two files whoselocations and/or contents must be maintained consistently, although thedisadvantages are reduced with respect to the embodiment of FIG. 5D.

Yet another embodiment, and in this instance the presently preferredembodiment, is illustrated in FIG. 5F. As illustrated therein, theVisual Link Dataset 500 is appended to an otherwise unaltered GraphicIcon 144 picture to form a single File 516, which may typically be animage file, wherein a Length-of-Dataset Number 518A is appended to theVisual Link Dataset 500 in the single File 516 and represents thelocation of the beginning of the Visual Link Dataset 500 in the File 516relative to the end of the File 516, and thereby the end of the GraphicIcon 144 picture in the File 516 relative to the end of the File 516.The beginning of the Visual Link Dataset 500 is thereby found by seekingto end-of-file minus size of Length-of-Dataset Number 518A, interpretingthe Length-of-Dataset Number 518A found at that location, and seekingbackward this number of bytes to find the beginning of the Visual LinkDataset 500 component, and the end of the Graphic Icon 144 picturecomponent.

In the present embodiment, Length-of-Dataset Number 518A is stored as afixed number of ASCII characters, such as four, and high digit first,rather than as an n-byte number, thereby avoiding possible confusionthat could arise from differences in byte order of words and doublewords, and differences in bit order in a byte, in different systems. Inyet other implementations of the embodiment shown in FIG. 5F,Length-of-Dataset Number 518A may be accompanied by a URL Pointer 518Bidentifying the location of the URL 128 Designator 502 in the file andan END Pointer 518C identifying the location of the END 504= Designator502 in the file.

It will be apparent that the advantages of this embodiment are thatthere is only one file to maintain, move or copy and that the entirefile can be used for most purposes relevant to the display of anon-augmented picture file, such as a display on a screen. It should benoted, however, that if Graphic Icons 144 are dealt with only byprograms designed specifically for this purpose, this problem should notarise.

Possible disadvantages could arise in this embodiment due to thestructure of the file wherein the actual length of the composite file isdifferent from the length of the file as may be declared within theimage file subpart as the user or program constructing this part hasgenerally not anticipated an appendment. That is, the declared length ofthe composite file would typically be based upon the size of onecomponent of the file, such as the Graphic Icon 144 component of thefile if the composite file is treated as a normal picture file. If it isoperated upon by a utility that operates upon the file based upon thedeclared and expected end-of-file length, the utility may read andoperate upon the expected rather than actual length of the file. As aconsequence, portions of the file, such as the Visual Link Dataset 500component, may not be operated upon and may, for example, be truncatedin a copy or move operation. The same problem may arise when a person orprogram copies the file into a buffer that accommodates only theself-declared size of the picture file as declared in the image part ofthe file.

In a related potential problem, any editor that is selected for use inmodifying the file must be chosen so as to not make end of line or filemodifications or other conversions that might alter the data stored in,for example, the Graphic Icon 144 picture component. In this instance,it may be preferable that the Visual Link Dataset 500 component of thecomposite file be split off from the composite for editing, and the twofiles subsequently re-concatenated.

It will be noted, however, that a system's file system will generallyknow or recognize the actual complete size or length of the file and itis only the length indicator or indicators within the image part of thefile that may give misleading indications of the length of the compositefile. Programs or utilities that do not rely only on the self declaredlength of the image part of the file will therefore generally avoid suchproblems.

In a fourth alternate embodiment, many bitmap picture file protocols,such as those for JPEG and GIF, accommodate application specificextensions or inclusions and, in these instance, the Visual Link Dataset500 could be added to the Graphic Icon 144 image file as an applicationspecific extension. This embodiment is advantageous in corresponding toestablished standards of the field of art and in relaxing or avoiding atleast some of the caveats discussed above in appending a Visual LinkDataset 500 component to an image file. A disadvantage, however, is thatutility programs for dealing with the Visual Link Dataset 500 componentsof the resulting composite files must be custom-designed or adjusted toeach such image file format.

Finally, those of skill in the relevant arts will appreciate that thereare still further methods for associating the two files. For example, acommon basename with different extensions, such as ‘elephant.gif’ and‘elephant.txt’, would have essentially the advantages and disadvantagesdiscussed above with regard to the embodiment illustrated in FIG. 5E.

7. Hash Protection of Files Such as Visual Links 142 (FIGS. 6A and 6B)

Referring to FIG. 6A, therein is represented a Hashing ProtectionMechanism 600 implemented, for example, in a Local System 112 or aServer System 114 for encoding computer system files, such as a VisualLink 142 file, to allow detection of the unauthorized construction ormodification of such files.

Hashing Protection Mechanism 600 performs this finction for a File 602by reading a Encrypting Part 604 of the Data 606 of the File 602,wherein Encrypting Part 604 comprises at least a part of Data 606, andgenerating a “Hash” Encrypted Value 608 by performing a known hashingoperation on Encrypting Part 604. Hashing Protection Mechanism 600 thenassociates the Hash Encrypted Value 608 with the File 602, for example,by embedding Hash Encrypted Value 608 as a Stored Hash Encrypted Value608S in a Storing Part 610 of Data 606 that is not included or containedwithin Encrypting Part 604. For these purposes, and as illustrated inFIG. 6A, Hashing Protection Mechanism 600 includes a File Reader 612 foridentifying and reading an Encrypting Part 604 of a File 602, a HashingMechanism 614 for performing the hashing operation on the EncryptingPart 604, and a File Writer 616 for writing Stored Hash Encrypted Value608S into Storing Part 610.

Thereafter, Hashing Protection Mechanism 600 may determine whether therehas been an unauthorized construction or modification of a File 602 hasbeen altered by reading Encrypting Part 604, performing the hashingoperation on Encrypting Part 604 to determine a Confirmation HashEncrypted Value 608C, and comparing the Confirmation Hash EncryptedValue 608C with the Stored Hash Encrypted Value 608S from Storing Part610. For these purposes, and as illustrated in FIG. 6A, HashingProtection Mechanism 600 again includes File Reader 612 for readingEncrypting Part 604 and Hashing Mechanism 614 for generating aConfirmation Hash Encrypted Value 608C from Entrypting Part 604. HashingProtection Mechanism 600 further includes Comparator 618 for reading theStored Hash Encrypted Value 6908S from Storing Part 610 comparing theStored Hash Encrypted Value 608S and the Confirmation Hash EnctryptedValue 608C to determine whether the File 602 has be properly constructedor modified by an authorized user or system.

It will be apparent that, because the specific hashing algorithm used inthe hashing operation is known or available only to users or systemsauthorized to construct or modify Files 602, only an authorized user orsystem will be able to generate a Confirmation Hash Encrypted Value 608Cfrom Encrypting Part 604 that is equal to the Stored Hash EncryptedValue 608S. If the values are equal, then the File 602 has not beenconstructed or modified without proper authorization and, if the valuesare not equal then the File 602 has been constructed or modified with ahashing algorithm different from that available and known to theauthorized users or systems. This method assumes, of course, that thehashing algorithm is held in secrecy, which is a common requirement foralmost all protection mechanisms, and that the algorithm is sufficientlycomplex to avoid accidental or easy discovery, which is a generalrequirement of all protection mechanisms and a common characteristic ofmany hashing algorithms.

According to the present invention, Storing Part 610 in which StoredHash Encrypted Value 608C is stored is comprised of portions of Data 606that are not only separate from Encrypting Part 604 but also arecharacterized by having alternate forms of expression, that is,different states, that are semantically equivalent to a user or acomputer system, and wherein the alternate, semantically equivalentstates of the Data 606 in Storing Part 610 are used to encode the StoredHash Encrypted Value 608C.

For example, there may be portions of Data 606 comprised of legiblewords or pseudo-words or abbreviations in English text that, for theirprimary purpose are semantically equivalent to a human user, such as“END” and “end”, or that are treated in a case insensitive manner by asystem. Assuming, for example, that a Stored Hash Encrypted Value 608Cis comprised of or expressed as a binary string, such as “101”, and thatan upper case letter is used to indicate a “1” while a lower case letteris used to represent a “0”, this Stored Hash Encrypted Value 608C couldbe encoded in the word “end” as “EnD” so long as this expression of“end” was, as necessary for its primary purpose, semantically equivalentto a user or that a system treated the word in a case insensitivemanner. Thus, “end”, “End”, “eNd”, “enD”, “ENd”, EnD”, eND”, and “END”could all be used to encode different Stored Hash Encrypted Values 608Cif these varients of “end” had equivalent meanings to a user or asystem, as necessary for the primary purpose of the word.

Therefore, minute differences in ASCII code, for example, may signifybits of a Stored Hash Encrypted Value 608C, with one bit of the StoredHash Encrypted Value 608C being represented by each ASCII character orstring of identically treated ASCII characters, so long as the use ofalternate versions of the characters or strings of characters did notinterfere with the primary functions of the characters. Also, and asdescribed above, these characters or strings of characters may not orshould not be used as the Encrypting Part 604. In addition, the parts ofa file that are used for Encrypting Part 604 and for Storing Part 610should be easily identifiable and locatable by a system.

This method of the present invention for encoding a hash encrypted valuegenerated from portions of the data of a file and stored in otherportions of the data of the file having alternate but semanticallyequivalent expressions that may be used to encode the hash encryptedvalued is illustrated in FIGS. 6B-1 and 6B-2 wherein FIG. 6B-1represents an original file and FIG. 6B-2 represents the same file withthe bits of the hash encrypted value, which are listed in a column atthe left of the file, encoded in the corresponding case-distinguishingbits of the ASCII characters used to encode the hash encrypted value. Itwill be noted that the substrings of characters that are considered asvalid for encoding by alternate expression are all contiguous substringsof letters terminated by the “=” sign. It will also be noted that, forcosmetic reasons, in each such substring all characters but the firstshare the same change, that is, the same alternate form of expressionand collectively represent one bit of the hash encrypted value, and thatthe beginning character of each substring is left in the upper casestate.

It may be apparent, therefore, that the hashing protection method andmechanism of the present invention is no more complex and requires nomore effort, for example, in maintaining the secrecy of the hashingalgorithm, than alternate protection mechanisms. It is further apparentthat the hashing protection method and mechanism of the presentinvention is advantageous in that the encrypted hash value is stored ina non-intrusive manner that takes no extra space in a file and does notchange the encrypted hash value of the file and in a manner that doesnot impede human or system interpretation of the data in the file.

Finally, while the invention has been particularly shown and describedwith reference to preferred embodiments of the apparatus and methodsthereof, it will be also understood by those of ordinary skill in theart that various changes, variations and modifications in form, detailsand implementation may be made therein without departing from the spiritand scope of the invention as defined by the appended claims. Therefore,it is the object of the appended claims to cover all such variation andmodifications of the invention as come within the true spirit and scopeof the invention.

What is claimed is:
 1. For use in a data processing system including amemory for storing programs and files containing data and a processorfor operating on the data under control of the programs, a captureengine for extracting graphics information from a data file andgenerating a corresponding graphic icon forming a displayable imagerepresenting the graphics information, comprising: a grayscale imagegenerator for receiving an original image and generating a correspondinggrayscale image containing brightness values representing the originalimage, an edge image mechanism for receiving the grayscale image andgenerating a corresponding edge image representing areas of visuallysignificant graphic structure in the grayscale image as represented byareas of change in the brightness values, a candidate region searchmechanism for receiving the edge image and identifying initial candidateregions of the edge image representing visually significant areas of theoriginal image, and a candidate region adjustment and comparatormechanism for selecting a candidate region to be used in generating acorresponding graphic icon and for adjusting the selected candidateregion to conform to predetermined dimensions for a graphic icon.
 2. Thecapture engine of claim 1, wherein the grayscale image generatorcomprises: a grayscale lookup table for storing grayscale brightnessvalues corresponding to each of the possible brightness values of imageelements in original images, a grayscale converter for reading thebrightness values of the image elements of the original image, readingthe corresponding grayscale brightness values stored in the grayscalelookup table, and generating a corresponding grayscale image whereineach image element of the original image is represented by a grayscaleimage element having the grayscale brightness value corresponding to thebrightness value of the image element.
 3. The capture engine of claim 2,wherein each image element of the original image is represented by a setof color values, the set of color values of an image element of theoriginal element representing the color and brightness value of theimage element, and wherein: the grayscale lookup table contains agrayscale brightness value corresponding to each of the possible sets ofcolor values of image elements in original images, and the grayscaleconverter is responsive to the set of color values of each image elementof the original image for reading the corresponding grayscale brightnessvalues stored in the grayscale lookup table, and generating acorresponding grayscale image wherein each image element of the originalimage is represented by a grayscale image element having the grayscalebrightness value corresponding to the set of color values of the imageelement.
 4. The capture engine of claim 2, wherein: the grayscale imagegenerator generates each image element of the grayscale image from aplurality of corresponding elements of the original image so that thenumber of image elements in the grayscale image is proportionallyreduced from the number of image elements in the corresponding originalimage.
 5. The capture engine of claim 1, wherein the edge imagemechanism comprises: an edge image generator for receiving an inputgrayscale image and generating a corresponding output edge image whereineach image element of an output edge image is represented by an edgevalue proportionate to the difference between the brightness value ofthe corresponding image element in the input grayscale image and thebrightness values of the neighboring image elements of the correspondingimage element in the input grayscale image, wherein an output edge imagethereby represents areas of visually significant graphic structure inthe input grayscale image as represented by areas of change in thebrightness values of the input grayscale image.
 6. The capture engine ofclaim 5, wherein the edge image mechanism further comprises: a lowpassfilter connected between the grayscale image generator and the edgeimage generator for receiving a grayscale image and generating acorresponding filtered grayscale image of reduced resolution andprovides the filtered grayscale image to the edge image generator as thecorrespond input grayscale image to the edge image generator, whereineach image element of a filtered grayscale image is determined as aproportioned summation of the brightness values of selected neighboringimage elements of the image element of the grayscale image, so that afiltered grayscale image thereby represents visual structures ofsignificant extents of the grayscale image.
 7. The capture engine ofclaim 5, wherein: the edge image generator generates each image elementof an output edge image from a plurality of corresponding elements ofthe input grayscale image so that the number of image elements in theoutput edge image is proportionally reduced from the number of imageelements in the corresponding grayscale input image.
 8. The captureengine of claim 1, wherein the candidate region search mechanismcomprises: a rectangular sum array generator for receiving an outputedge image and generating a corresponding rectangular sum array having asum element corresponding to each image element of the output edge imagewherein each sum element represents the sum of the edge values of theimage elements of a region of the output edge image bounded by acoordinate location of the output edge image and the output edge imageelement corresponding to the sum element, and a rectangular sum searchmechanism for examining the summed edge values represented in each ofthe sum elements of the rectangular sum array and identifying at leastone initial candidate region having a high summed edge value, eachinitial candidate region thereby representing and corresponding to aregion of the original image having a significant visual structure. 9.The capture engine of claim 1, wherein the candidate region adjustmentand comparator mechanism comprises: an edge adjuster for receiving theat least one initial candidate region and generating from each initialcandidate region a corresponding grown candidate region by adjusting theextents of each initial candidate region to include all portions ofsignificant visual structures having a part thereof contained within theinitial candidate region, generating from each grown candidate region acorresponding edge adjusted candidate region by adjusting the extents ofeach grown candidate region so that each edge adjusted candidate regionconforms to one of a set of predetermined extents for graphic icons,comparing each edge adjusted candidate region with others of the edgeadjusted candidate regions and selecting the edge adjusted candidateregion having the highest amount of significant visual structure,comparing each selected edge adjusted candidate region with previouslyexisting graphic icons and, when the selected edge adjusted candidateregion is similar to a previously existing graphic icon, selecting anext one of the at least one edge adjusted candidate regions, and, whena selected candidate region is distinguished from the previouslyexisting graphic icons, identifying the corresponding region of theoriginal image for use as a graphic icon.
 10. For use in a dataprocessing system including a memory for storing programs and filescontaining data and a processor for operating on the data under controlof the programs, a capture engine for extracting graphics informationfrom a data file and generating a corresponding graphic icon forming adisplayable image representing the graphics information, comprising: agrayscale image generator for receiving an original image and generatinga corresponding grayscale image containing brightness valuesrepresenting the original image, including a grayscale lookup table forstoring grayscale brightness values corresponding to each of thepossible brightness values of image elements in original images, agrayscale converter for reading the brightness values of the imageelements of the original image, reading the corresponding grayscalebrightness values stored in the grayscale lookup table, and generating acorresponding grayscale image wherein each image element of the originalimage is represented by a grayscale image element having the grayscalebrightness value corresponding to the brightness value of the imageelement, an edge image mechanism for receiving the grayscale image andgenerating a corresponding edge image representing areas of visuallysignificant graphic structure in the grayscale image as represented byareas of change in the brightness values, including an edge imagegenerator for receiving an input grayscale image and generating acorresponding output edge image wherein each image element of an outputedge image is represented by an edge value proportionate to thedifference between the brightness value of the corresponding imageelement in the input grayscale image and the brightness values of theneighboring image elements of the corresponding image element in theinput grayscale image, wherein an output edge image thereby representsareas of visually significant graphic structure in the input grayscaleimage as represented by areas of change in the brightness values of theinput grayscale image, and a candidate region search mechanism forreceiving the edge image and identifying initial candidate regions ofthe edge image representing visually significant areas of the originalimage, including a rectangular sum array generator for receiving anoutput edge image and generating a corresponding rectangular sum arrayhaving a sum element corresponding to each image element of the outputedge image wherein each sum element represents the sum of the edgevalues of the image elements of a region of the output edge imagebounded by a coordinate location of the output edge image and the outputedge image element corresponding to the sum element, and a rectangularsum search mechanism for examining the summed edge values represented ineach of the sum elements of the rectangular sum array and identifying atleast one initial candidate region having a high summed edge value, eachinitial candidate region thereby representing and corresponding to aregion of the original image having a significant visual structure, anda candidate region adjustment and comparator mechanism for selecting acandidate region to be used in generating a corresponding graphic iconand for adjusting the selected candidate region to conform topredetermined dimensions for a graphic icon.
 11. The capture engine ofclaim 10, wherein each image element of the original image isrepresented by a set of color values, the set of color values of animage element of the original element representing the color andbrightness value of the image element, and wherein: the grayscale lookuptable contains a grayscale brightness value corresponding to each of thepossible sets of color values of image elements in original images, andthe grayscale converter is responsive to the set of color values of eachimage element of the original image for reading the correspondinggrayscale brightness values stored in the grayscale lookup table, andgenerating a corresponding grayscale image wherein each image element ofthe original image is represented by a grayscale image element havingthe grayscale brightness value corresponding to the set of color valuesof the image element.
 12. The capture engine of claim 10, wherein: thegrayscale image generator generates each image element of the grayscaleimage from a plurality of corresponding elements of the original imageso that the number of image elements in the grayscale image isproportionally reduced from the number of image elements in thecorresponding original image.
 13. The capture engine of claim 10,wherein the edge image mechanism further comprises: a lowpass filterconnected between the grayscale image generator and the edge imagegenerator for receiving a grayscale image and generating a correspondingfiltered grayscale image of reduced resolution and provides the filteredgrayscale image to the edge image generator as the correspond inputgrayscale image to the edge image generator, wherein each image elementof a filtered grayscale image is determined as a proportioned summationof the brightness values of selected neighboring image elements of theimage element of the grayscale image, so that a filtered grayscale imagethereby represents visual structures of significant extents of thegrayscale image.
 14. The capture engine of claim 12, wherein: the edgeimage generator generates each image element of an output edge imagefrom a plurality of corresponding elements of the input grayscale imageso that the number of image elements in the output edge image isproportionally reduced from the number of image elements in thecorresponding grayscale input image.
 15. The capture engine of claim 10,wherein the candidate region adjustment and comparator mechanismcomprises: an edge adjuster for receiving the at least one initialcandidate region and generating from each initial candidate region acorresponding grown candidate region by adjusting the extents of eachinitial candidate region to include all portions of significant visualstructures having a part thereof contained within the initial candidateregion, generating from each grown candidate region a corresponding edgeadjusted candidate region by adjusting the extents of each growncandidate region so that each edge adjusted candidate region conforms toone of a set of predetermined extents for graphic icons, comparing eachedge adjusted candidate region with others of the edge adjustedcandidate regions and selecting the edge adjusted candidate regionhaving the highest amount of significant visual structure, comparingeach selected edge adjusted candidate region with previously existinggraphic icons and, when the selected edge adjusted candidate region issimilar to a previously existing graphic icon, selecting a next one ofthe at least one edge adjusted candidate regions, and, when a selectedcandidate region is distinguished from the previously existing graphicicons, identifying the corresponding region of the original image foruse as a graphic icon.